Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with...

39
Kubernetes on Oracle Cloud Infrastructure Overview and Manual Deployment Guide ORACLE WHITE PAPER | FEBRUARY 2018

Transcript of Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with...

Page 1: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

Kubernetes on Oracle Cloud Infrastructure

Overview and Manual Deployment Guide

O R A C L E W H I T E P A P E R | F E B R U A R Y 2 0 1 8

2 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Disclaimer

The following is intended to outline our general product direction It is intended for information

purposes only and may not be incorporated into any contract It is not a commitment to deliver any

material code or functionality and should not be relied upon in making purchasing decisions The

development release and timing of any features or functionality described for Oraclersquos products

remains at the sole discretion of Oracle

3 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Table of Contents

Target Audience 4

Introduction 4

Overview of Kubernetes 4

Container Images 5

Application Deployment 5

Container Orchestration Features 6

Cluster Architecture for Kubernetes Manual Deployment on Oracle Cloud Infrastructure 7

etcd 8

Kubernetes Masters 8

Kubernetes Workers 9

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure 9

Step 1 Create Oracle Cloud Infrastructure Resources 9

Step 2 Set Up a Certificate Authority and Create TLS Certificates 13

Step 3 Bootstrap an HA etcd Cluster 19

Step 4 Set Up RBAC 21

Step 5 Bootstrap an HA Kubernetes Control Plane 22

Step 6 Add a Worker 27

Step 7 Configure kubectl for Remote Access 34

Step 8 Deploy Kube-DNS 35

Step 9 Smoke Test 35

Appendix A Security Rules 37

4 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Target Audience

This document is intended for customers who are interested in learning how Kubernetes works by

deploying it on Oracle Cloud Infrastructure or who have considerable experience with Kubernetes

and want a baseline deployment process in order to create their own highly configured clusters

This document assumes that the user is working from a computer running a macOS operating

system and has an understanding of UNIX commands and file systems

Users of this document should also be familiar with the fundamentals of the Oracle Cloud

Infrastructure For information go to httpsdocsus-phoenix-1oraclecloudcom If this is the first

time that you have used the platform we recommend specifically the tutorial at httpsdocsus-

phoenix-1oraclecloudcomContentGSGReferenceoverviewworkflowhtm

Introduction

Kubernetes is the most popular container orchestration tool available today Although the

Kubernetes open-source project is still young and experiencing tremendous growth when it is

deployed properly Kubernetes can be a reliable tool for running container workloads in production

This document presents a starting point for deploying a secure highly available Kubernetes cluster

on Oracle Cloud Infrastructure The cluster created from the instructions in this document might be

sufficient for your needs However if you want to configure your cluster beyond what is presented

here yoursquoll need supplementary materials and white papers to address the various customization

options and updates to Kubernetes

Kubernetes is an incredibly fast-moving project with frequent new releases and bug fixes

Accordingly this document addresses deploying Kubernetes version 16 (the stable build at the

time of writing) Future documents will address the process to upgrade the cluster made in this

document

Overview of Kubernetes

This section provides a brief introduction to Kubernetes If you are already familiar with

Kubernetes you can skip this section

Kubernetes is the most popular container orchestration tool available and is maintained by one of

the fastest-growing open-source communities The Kubernetes project originated within Google a

long-time user of massive numbers of containers To manage these containers well they needed

to develop a system for container orchestration Kubernetes combines the lessons that Google

5 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

learned from years of container usage into a single tool with an array of features that make

container orchestration simple and adaptable to the wide variety of use cases in the technology

industry Since it became open source in July 2015 the capabilities of Kubernetes have continued

to grow Issues and new feature requests are tracked on the public GitHub project with new major

versions released approximately every two months

Containers are designed to solve problems with traditional application deployment such as

missing dependencies when installing an application or trouble deploying applications on specific

OS versions Container orchestrators aim to solve problems with scaling these applications

Container Images

The application with all of its dependencies is kept in a container image When run by a container

engine such as Docker the container image runs as a container The process of creating this

container image for an application is known as containerization Containerization is beneficial in

cases where the application would interact poorly with other applications if deployed on the same

machine The container provides a level of isolation that although not fully multi-tenant can

prevent applications from causing problems with other applications running on the same physical

host For example containers simplify Java application deployment by bundling the applicationrsquos

dependencies (like the specific Java Runtime Environment version) with the application in the

container Additionally if the application runs on Linux the container also abstracts the flavor and

version of the Linux OS All dependencies required to run the application residing in the OS can

also be bundled in the container As a result a containerized application runs the same on Oracle

Linux as it would on Ubuntu

Application Deployment

After the application and its dependencies are bundled in a container image the next step is to

distribute that application to all the groups that need it Assume that many teams need to use this

application and that the scale of the teamsrsquo usage might change over time In a traditional data

center scaling this application would likely require an analysis to estimate the resources needed to

run this application across the company for the next quarter or perhaps the entire next fiscal year

The IT organization that manages these physical resources would need to order new equipment to

satisfy the needs of the business In the cloud new resources on which to run the application can

be acquired on-demand and in a greater variety of sizes (that is a virtual machine with fewer cores

rather than a whole physical machine) however the organization still needs to manage application

deployments to those resources and manage those deployments to respond to the needs of the

business over time Container orchestration simplifies the solution to this problem the

containerization of the application makes the application easy to deploy in a variety of

6 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

environments and the container orchestrator provides a way to manage and scale that application

as needed

Container Orchestration Features

Kubernetes provides several features to accomplish the task of orchestrating containerized

applications

Declarative Management

Kubernetes is designed to be managed in a declarative manner rather than an imperative manner

An example of imperative management is installing a program via the apt-get install

command which imperatively installs the application in a certain location When an update to the

program is needed you imperatively tell apt-get to update the application

Declarative management is managing based on desired state Deployments are a Kubernetes

component designed to hold the state of an application Defined in JSON or YAML the

Deployment contains information about the container image such as its version the number of

containers that should exist within the Kubernetes cluster and a variety of other properties that

Kubernetes needs in order to deploy the application properly You simply state the number of

application instances that you want to have and Kubernetes creates and maintains the necessary

number of containers across your resources If a container fails for any reason (for example the

virtual machine on which the container is running goes offline) Kubernetes automatically stands

up a new one on a healthy node to maintain the state declared in the Deployment Kubernetes

constantly checks the actual state against the desired state and ensures that the two states match

The declarative model provides a benefit over the imperative model in which the system must

always be told what to do In the declarative model Kubernetes does the work for you as long as

the definitions for the desired system state exists

7 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Rolling Upgrades and Rollbacks

Another benefit of deploying applications via Kubernetes Deployments is its built-in rolling upgrade

and rollback functionality For example when a YAML file that defines the state of a Deployment is

updated with a new version of the container image Kubernetes recognizes this change and begins

to shut down existing instances of the older version while creating new instances with the updated

version While Kubernetes performs this rolling upgrade it continues to direct requests to running

container instances which normally results in zero downtime Likewise if there is an issue with the

upgrade Kubernetes performs a rollback as needed

Load Balancer

To manage connections to the applications deployed as containers within Kubernetes the

Kubernetes Service component provides a software-defined load balancer within the Kubernetes

cluster For example a deployment of three instances of the Java application might be accessed

by means of a single point in the Kubernetes Service As a result if an instance becomes

unavailable or if a rolling upgrade is performed the application can still be accessed without

interruption

These basic components of Kubernetes make it an excellent way to orchestrate large numbers of

containerized applications across a pool of resources As you consider how best to use container

orchestration to meet your needs learn more about Kubernetes by reading their documentation at

httpskubernetesiodocs

Cluster Architecture for Kubernetes Manual Deployment on Oracle Cloud Infrastructure

This section describes the components that make up a functioning Kubernetes cluster and

explains how those components are deployed across the compute resources that make up the

cluster The architecture described in this section will be deployed in the ldquoKubernetes Manual

Deployment Guide for Oracle Cloud Infrastructurerdquo (the next section)

A Kubernetes cluster consists of three main components etcd Kubernetes masters (or

controllers) and Kubernetes workers (or nodes) This guide explains how to create a highly

available (HA) Kubernetes cluster with the following architecture

Three etcd nodes (across three availability domains in one region)

Three Kubernetes masters (or controllers) (across three availability domains in one

region)

Three Kubernetes workers (or nodes) (across three availability domains in one region)

8 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The steps in this guide produce an infrastructure similar to the one shown in the following diagram

The etcd cluster is configured to run on a separate set of compute resources from the Kubernetes

cluster

The following sections explain the components in greater detail

etcd

etcd is a key-value store created by CoreOS Kubernetesrsquo state information is stored in the etcd

cluster This should not be confused with running an etcd cluster via Kubernetes rather this etcd

cluster is helping to run Kubernetes itself

In this guide the etcd cluster is configured to run on a separate set of compute resources from the

Kubernetes cluster Running etcd on separate compute resources provides greater isolation

between etcd and the components of the Kubernetes cluster

Kubernetes Masters

The Kubernetes masters (or controllers) are machines (virtual or physical) that run the API server

controller manager and scheduler components of the Kubernetes cluster

9 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Kubernetes Workers

The Kubernetes workers (or nodes) are machines (virtual or physical) that run the kubelet

component of the Kubernetes cluster The workers are the resources on which Kubernetes

schedules containers (or pods)

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure

This guide explains how to deploy and configure all the components and features required to run

Kubernetes on Oracle Cloud Infrastructure This guide assumes that you are starting with a ldquoclean

slaterdquo environment during the course of this guide you will create all the resources that you need

in Oracle Cloud Infrastructure

This guide walks through the following tasks

1 Create Oracle Cloud Infrastructure resources

2 Generate certificates for Kubernetes components

3 Bootstrap an HA etcd cluster

4 Generate token and configuration files to authenticate components

5 Bootstrap the HA Kubernetes masters (Kubernetes control plane)

6 Add a worker

7 Configure remote access

8 Deploy Kube-DNS

9 Smoke test

Step 1 Create Oracle Cloud Infrastructure Resources

This guide does not explain how to create the Oracle Cloud Infrastructure resources that you need

to create the Kubernetes cluster This section lists the required resources but you must create

them on your own by using the Oracle Cloud Infrastructure Console CLI API SDKs or the

Terraform provider For instructions see the Oracle Cloud Infrastructure documentation

Networking

To create the Kubernetes cluster in this guide you need the following Networking components

For more information about creating Networking components see the Networking section of the

Oracle Cloud Infrastructure documentation

10 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

VCN

You need a single virtual cloud network (VCN) with the following values

VCN Name CIDR Description

k8sOCIoraclevcncom 1000016 VCN used to host network resources for the

Kubernetes cluster

Subnets

You need a VCN and at least one subnet per availability domain (three subnets total) In a

production configuration we recommend creating an etcd master and worker subnet per

availability domain which would result in nine subnets For a test or learning deployment intended

to be removed later creating three subnets (one per availability domain) is sufficient The following

values describe the recommended configuration for a cluster in a region with three availability

domains

Use the following values

Subnet Name CIDR Availability

Domain Description

publicETCDSubnetAD1sub 10020024 AD1 Subnet used for etcd host in AD1

publicETCDSubnetAD2sub 10021024 AD2 Subnet used for etcd host in AD2

publicETCDSubnetAD3sub 10022024 AD3 Subnet used for etcd host in AD3

publicK8SMasterSubnetAD1sub 10030024 AD1 Subnet used for Kubernetes

masters in AD1

publicK8SMasterSubnetAD2sub 10031024 AD2 Subnet used for Kubernetes

masters in AD2

publicK8SMasterSubnetAD3sub 10032024 AD3 Subnet used for Kubernetes

masters in AD3

publicK8SWorkerSubnetAD1sub 10040024 AD1 Subnet used to host Kubernetes

workers in AD1

publicK8SWorkerSubnetAD2sub 10041024 AD2 Subnet used to host Kubernetes

workers in AD2

publicK8SWorkerSubnetAD3sub 10042024 AD3 Subnet used to host Kubernetes

workers in AD3

11 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Security Lists

A production configuration should include the following security lists

etcd_security_list

k8sMaster_security_list

k8sWorker_security_list

For a list of the recommended security rules for each security list see ldquoAppendix A Security

Rulesrdquo

Internet Gateway and Route Table

Your configuration should include one internet gateway and one route table rule that allows your

Kubernetes cluster to access the internet through the internet gateway

The route table rule should have a destination CIDR block of 00000 and target type of

internet gateway The target should be the internet gateway that you intend to use with your

Kubernetes cluster

Load Balancer

The recommended cluster configuration requires two load balancers Create a private load

balancer for your etcd nodes and a public load balancer for your Kubernetes masters Populate

the following table with your load balancerrsquos information to refer to throughout the guide

In the guide the public IP address of your Kubernetes master load balancer is referred to as

loadbalancer_public_ip

Use the following values

Load Balancer

Name

Load

Balancer

Type

Subnet1 Subnet2 Public IP Private IP

Compute

Resource

Back Ends

lb-etcd Private Not applicable Etcd1

Etcd2

Etcd3

lb-k8smaster Public KubeM1

KubeM2

KubeM3

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 2: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

2 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Disclaimer

The following is intended to outline our general product direction It is intended for information

purposes only and may not be incorporated into any contract It is not a commitment to deliver any

material code or functionality and should not be relied upon in making purchasing decisions The

development release and timing of any features or functionality described for Oraclersquos products

remains at the sole discretion of Oracle

3 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Table of Contents

Target Audience 4

Introduction 4

Overview of Kubernetes 4

Container Images 5

Application Deployment 5

Container Orchestration Features 6

Cluster Architecture for Kubernetes Manual Deployment on Oracle Cloud Infrastructure 7

etcd 8

Kubernetes Masters 8

Kubernetes Workers 9

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure 9

Step 1 Create Oracle Cloud Infrastructure Resources 9

Step 2 Set Up a Certificate Authority and Create TLS Certificates 13

Step 3 Bootstrap an HA etcd Cluster 19

Step 4 Set Up RBAC 21

Step 5 Bootstrap an HA Kubernetes Control Plane 22

Step 6 Add a Worker 27

Step 7 Configure kubectl for Remote Access 34

Step 8 Deploy Kube-DNS 35

Step 9 Smoke Test 35

Appendix A Security Rules 37

4 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Target Audience

This document is intended for customers who are interested in learning how Kubernetes works by

deploying it on Oracle Cloud Infrastructure or who have considerable experience with Kubernetes

and want a baseline deployment process in order to create their own highly configured clusters

This document assumes that the user is working from a computer running a macOS operating

system and has an understanding of UNIX commands and file systems

Users of this document should also be familiar with the fundamentals of the Oracle Cloud

Infrastructure For information go to httpsdocsus-phoenix-1oraclecloudcom If this is the first

time that you have used the platform we recommend specifically the tutorial at httpsdocsus-

phoenix-1oraclecloudcomContentGSGReferenceoverviewworkflowhtm

Introduction

Kubernetes is the most popular container orchestration tool available today Although the

Kubernetes open-source project is still young and experiencing tremendous growth when it is

deployed properly Kubernetes can be a reliable tool for running container workloads in production

This document presents a starting point for deploying a secure highly available Kubernetes cluster

on Oracle Cloud Infrastructure The cluster created from the instructions in this document might be

sufficient for your needs However if you want to configure your cluster beyond what is presented

here yoursquoll need supplementary materials and white papers to address the various customization

options and updates to Kubernetes

Kubernetes is an incredibly fast-moving project with frequent new releases and bug fixes

Accordingly this document addresses deploying Kubernetes version 16 (the stable build at the

time of writing) Future documents will address the process to upgrade the cluster made in this

document

Overview of Kubernetes

This section provides a brief introduction to Kubernetes If you are already familiar with

Kubernetes you can skip this section

Kubernetes is the most popular container orchestration tool available and is maintained by one of

the fastest-growing open-source communities The Kubernetes project originated within Google a

long-time user of massive numbers of containers To manage these containers well they needed

to develop a system for container orchestration Kubernetes combines the lessons that Google

5 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

learned from years of container usage into a single tool with an array of features that make

container orchestration simple and adaptable to the wide variety of use cases in the technology

industry Since it became open source in July 2015 the capabilities of Kubernetes have continued

to grow Issues and new feature requests are tracked on the public GitHub project with new major

versions released approximately every two months

Containers are designed to solve problems with traditional application deployment such as

missing dependencies when installing an application or trouble deploying applications on specific

OS versions Container orchestrators aim to solve problems with scaling these applications

Container Images

The application with all of its dependencies is kept in a container image When run by a container

engine such as Docker the container image runs as a container The process of creating this

container image for an application is known as containerization Containerization is beneficial in

cases where the application would interact poorly with other applications if deployed on the same

machine The container provides a level of isolation that although not fully multi-tenant can

prevent applications from causing problems with other applications running on the same physical

host For example containers simplify Java application deployment by bundling the applicationrsquos

dependencies (like the specific Java Runtime Environment version) with the application in the

container Additionally if the application runs on Linux the container also abstracts the flavor and

version of the Linux OS All dependencies required to run the application residing in the OS can

also be bundled in the container As a result a containerized application runs the same on Oracle

Linux as it would on Ubuntu

Application Deployment

After the application and its dependencies are bundled in a container image the next step is to

distribute that application to all the groups that need it Assume that many teams need to use this

application and that the scale of the teamsrsquo usage might change over time In a traditional data

center scaling this application would likely require an analysis to estimate the resources needed to

run this application across the company for the next quarter or perhaps the entire next fiscal year

The IT organization that manages these physical resources would need to order new equipment to

satisfy the needs of the business In the cloud new resources on which to run the application can

be acquired on-demand and in a greater variety of sizes (that is a virtual machine with fewer cores

rather than a whole physical machine) however the organization still needs to manage application

deployments to those resources and manage those deployments to respond to the needs of the

business over time Container orchestration simplifies the solution to this problem the

containerization of the application makes the application easy to deploy in a variety of

6 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

environments and the container orchestrator provides a way to manage and scale that application

as needed

Container Orchestration Features

Kubernetes provides several features to accomplish the task of orchestrating containerized

applications

Declarative Management

Kubernetes is designed to be managed in a declarative manner rather than an imperative manner

An example of imperative management is installing a program via the apt-get install

command which imperatively installs the application in a certain location When an update to the

program is needed you imperatively tell apt-get to update the application

Declarative management is managing based on desired state Deployments are a Kubernetes

component designed to hold the state of an application Defined in JSON or YAML the

Deployment contains information about the container image such as its version the number of

containers that should exist within the Kubernetes cluster and a variety of other properties that

Kubernetes needs in order to deploy the application properly You simply state the number of

application instances that you want to have and Kubernetes creates and maintains the necessary

number of containers across your resources If a container fails for any reason (for example the

virtual machine on which the container is running goes offline) Kubernetes automatically stands

up a new one on a healthy node to maintain the state declared in the Deployment Kubernetes

constantly checks the actual state against the desired state and ensures that the two states match

The declarative model provides a benefit over the imperative model in which the system must

always be told what to do In the declarative model Kubernetes does the work for you as long as

the definitions for the desired system state exists

7 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Rolling Upgrades and Rollbacks

Another benefit of deploying applications via Kubernetes Deployments is its built-in rolling upgrade

and rollback functionality For example when a YAML file that defines the state of a Deployment is

updated with a new version of the container image Kubernetes recognizes this change and begins

to shut down existing instances of the older version while creating new instances with the updated

version While Kubernetes performs this rolling upgrade it continues to direct requests to running

container instances which normally results in zero downtime Likewise if there is an issue with the

upgrade Kubernetes performs a rollback as needed

Load Balancer

To manage connections to the applications deployed as containers within Kubernetes the

Kubernetes Service component provides a software-defined load balancer within the Kubernetes

cluster For example a deployment of three instances of the Java application might be accessed

by means of a single point in the Kubernetes Service As a result if an instance becomes

unavailable or if a rolling upgrade is performed the application can still be accessed without

interruption

These basic components of Kubernetes make it an excellent way to orchestrate large numbers of

containerized applications across a pool of resources As you consider how best to use container

orchestration to meet your needs learn more about Kubernetes by reading their documentation at

httpskubernetesiodocs

Cluster Architecture for Kubernetes Manual Deployment on Oracle Cloud Infrastructure

This section describes the components that make up a functioning Kubernetes cluster and

explains how those components are deployed across the compute resources that make up the

cluster The architecture described in this section will be deployed in the ldquoKubernetes Manual

Deployment Guide for Oracle Cloud Infrastructurerdquo (the next section)

A Kubernetes cluster consists of three main components etcd Kubernetes masters (or

controllers) and Kubernetes workers (or nodes) This guide explains how to create a highly

available (HA) Kubernetes cluster with the following architecture

Three etcd nodes (across three availability domains in one region)

Three Kubernetes masters (or controllers) (across three availability domains in one

region)

Three Kubernetes workers (or nodes) (across three availability domains in one region)

8 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The steps in this guide produce an infrastructure similar to the one shown in the following diagram

The etcd cluster is configured to run on a separate set of compute resources from the Kubernetes

cluster

The following sections explain the components in greater detail

etcd

etcd is a key-value store created by CoreOS Kubernetesrsquo state information is stored in the etcd

cluster This should not be confused with running an etcd cluster via Kubernetes rather this etcd

cluster is helping to run Kubernetes itself

In this guide the etcd cluster is configured to run on a separate set of compute resources from the

Kubernetes cluster Running etcd on separate compute resources provides greater isolation

between etcd and the components of the Kubernetes cluster

Kubernetes Masters

The Kubernetes masters (or controllers) are machines (virtual or physical) that run the API server

controller manager and scheduler components of the Kubernetes cluster

9 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Kubernetes Workers

The Kubernetes workers (or nodes) are machines (virtual or physical) that run the kubelet

component of the Kubernetes cluster The workers are the resources on which Kubernetes

schedules containers (or pods)

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure

This guide explains how to deploy and configure all the components and features required to run

Kubernetes on Oracle Cloud Infrastructure This guide assumes that you are starting with a ldquoclean

slaterdquo environment during the course of this guide you will create all the resources that you need

in Oracle Cloud Infrastructure

This guide walks through the following tasks

1 Create Oracle Cloud Infrastructure resources

2 Generate certificates for Kubernetes components

3 Bootstrap an HA etcd cluster

4 Generate token and configuration files to authenticate components

5 Bootstrap the HA Kubernetes masters (Kubernetes control plane)

6 Add a worker

7 Configure remote access

8 Deploy Kube-DNS

9 Smoke test

Step 1 Create Oracle Cloud Infrastructure Resources

This guide does not explain how to create the Oracle Cloud Infrastructure resources that you need

to create the Kubernetes cluster This section lists the required resources but you must create

them on your own by using the Oracle Cloud Infrastructure Console CLI API SDKs or the

Terraform provider For instructions see the Oracle Cloud Infrastructure documentation

Networking

To create the Kubernetes cluster in this guide you need the following Networking components

For more information about creating Networking components see the Networking section of the

Oracle Cloud Infrastructure documentation

10 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

VCN

You need a single virtual cloud network (VCN) with the following values

VCN Name CIDR Description

k8sOCIoraclevcncom 1000016 VCN used to host network resources for the

Kubernetes cluster

Subnets

You need a VCN and at least one subnet per availability domain (three subnets total) In a

production configuration we recommend creating an etcd master and worker subnet per

availability domain which would result in nine subnets For a test or learning deployment intended

to be removed later creating three subnets (one per availability domain) is sufficient The following

values describe the recommended configuration for a cluster in a region with three availability

domains

Use the following values

Subnet Name CIDR Availability

Domain Description

publicETCDSubnetAD1sub 10020024 AD1 Subnet used for etcd host in AD1

publicETCDSubnetAD2sub 10021024 AD2 Subnet used for etcd host in AD2

publicETCDSubnetAD3sub 10022024 AD3 Subnet used for etcd host in AD3

publicK8SMasterSubnetAD1sub 10030024 AD1 Subnet used for Kubernetes

masters in AD1

publicK8SMasterSubnetAD2sub 10031024 AD2 Subnet used for Kubernetes

masters in AD2

publicK8SMasterSubnetAD3sub 10032024 AD3 Subnet used for Kubernetes

masters in AD3

publicK8SWorkerSubnetAD1sub 10040024 AD1 Subnet used to host Kubernetes

workers in AD1

publicK8SWorkerSubnetAD2sub 10041024 AD2 Subnet used to host Kubernetes

workers in AD2

publicK8SWorkerSubnetAD3sub 10042024 AD3 Subnet used to host Kubernetes

workers in AD3

11 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Security Lists

A production configuration should include the following security lists

etcd_security_list

k8sMaster_security_list

k8sWorker_security_list

For a list of the recommended security rules for each security list see ldquoAppendix A Security

Rulesrdquo

Internet Gateway and Route Table

Your configuration should include one internet gateway and one route table rule that allows your

Kubernetes cluster to access the internet through the internet gateway

The route table rule should have a destination CIDR block of 00000 and target type of

internet gateway The target should be the internet gateway that you intend to use with your

Kubernetes cluster

Load Balancer

The recommended cluster configuration requires two load balancers Create a private load

balancer for your etcd nodes and a public load balancer for your Kubernetes masters Populate

the following table with your load balancerrsquos information to refer to throughout the guide

In the guide the public IP address of your Kubernetes master load balancer is referred to as

loadbalancer_public_ip

Use the following values

Load Balancer

Name

Load

Balancer

Type

Subnet1 Subnet2 Public IP Private IP

Compute

Resource

Back Ends

lb-etcd Private Not applicable Etcd1

Etcd2

Etcd3

lb-k8smaster Public KubeM1

KubeM2

KubeM3

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 3: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

3 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Table of Contents

Target Audience 4

Introduction 4

Overview of Kubernetes 4

Container Images 5

Application Deployment 5

Container Orchestration Features 6

Cluster Architecture for Kubernetes Manual Deployment on Oracle Cloud Infrastructure 7

etcd 8

Kubernetes Masters 8

Kubernetes Workers 9

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure 9

Step 1 Create Oracle Cloud Infrastructure Resources 9

Step 2 Set Up a Certificate Authority and Create TLS Certificates 13

Step 3 Bootstrap an HA etcd Cluster 19

Step 4 Set Up RBAC 21

Step 5 Bootstrap an HA Kubernetes Control Plane 22

Step 6 Add a Worker 27

Step 7 Configure kubectl for Remote Access 34

Step 8 Deploy Kube-DNS 35

Step 9 Smoke Test 35

Appendix A Security Rules 37

4 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Target Audience

This document is intended for customers who are interested in learning how Kubernetes works by

deploying it on Oracle Cloud Infrastructure or who have considerable experience with Kubernetes

and want a baseline deployment process in order to create their own highly configured clusters

This document assumes that the user is working from a computer running a macOS operating

system and has an understanding of UNIX commands and file systems

Users of this document should also be familiar with the fundamentals of the Oracle Cloud

Infrastructure For information go to httpsdocsus-phoenix-1oraclecloudcom If this is the first

time that you have used the platform we recommend specifically the tutorial at httpsdocsus-

phoenix-1oraclecloudcomContentGSGReferenceoverviewworkflowhtm

Introduction

Kubernetes is the most popular container orchestration tool available today Although the

Kubernetes open-source project is still young and experiencing tremendous growth when it is

deployed properly Kubernetes can be a reliable tool for running container workloads in production

This document presents a starting point for deploying a secure highly available Kubernetes cluster

on Oracle Cloud Infrastructure The cluster created from the instructions in this document might be

sufficient for your needs However if you want to configure your cluster beyond what is presented

here yoursquoll need supplementary materials and white papers to address the various customization

options and updates to Kubernetes

Kubernetes is an incredibly fast-moving project with frequent new releases and bug fixes

Accordingly this document addresses deploying Kubernetes version 16 (the stable build at the

time of writing) Future documents will address the process to upgrade the cluster made in this

document

Overview of Kubernetes

This section provides a brief introduction to Kubernetes If you are already familiar with

Kubernetes you can skip this section

Kubernetes is the most popular container orchestration tool available and is maintained by one of

the fastest-growing open-source communities The Kubernetes project originated within Google a

long-time user of massive numbers of containers To manage these containers well they needed

to develop a system for container orchestration Kubernetes combines the lessons that Google

5 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

learned from years of container usage into a single tool with an array of features that make

container orchestration simple and adaptable to the wide variety of use cases in the technology

industry Since it became open source in July 2015 the capabilities of Kubernetes have continued

to grow Issues and new feature requests are tracked on the public GitHub project with new major

versions released approximately every two months

Containers are designed to solve problems with traditional application deployment such as

missing dependencies when installing an application or trouble deploying applications on specific

OS versions Container orchestrators aim to solve problems with scaling these applications

Container Images

The application with all of its dependencies is kept in a container image When run by a container

engine such as Docker the container image runs as a container The process of creating this

container image for an application is known as containerization Containerization is beneficial in

cases where the application would interact poorly with other applications if deployed on the same

machine The container provides a level of isolation that although not fully multi-tenant can

prevent applications from causing problems with other applications running on the same physical

host For example containers simplify Java application deployment by bundling the applicationrsquos

dependencies (like the specific Java Runtime Environment version) with the application in the

container Additionally if the application runs on Linux the container also abstracts the flavor and

version of the Linux OS All dependencies required to run the application residing in the OS can

also be bundled in the container As a result a containerized application runs the same on Oracle

Linux as it would on Ubuntu

Application Deployment

After the application and its dependencies are bundled in a container image the next step is to

distribute that application to all the groups that need it Assume that many teams need to use this

application and that the scale of the teamsrsquo usage might change over time In a traditional data

center scaling this application would likely require an analysis to estimate the resources needed to

run this application across the company for the next quarter or perhaps the entire next fiscal year

The IT organization that manages these physical resources would need to order new equipment to

satisfy the needs of the business In the cloud new resources on which to run the application can

be acquired on-demand and in a greater variety of sizes (that is a virtual machine with fewer cores

rather than a whole physical machine) however the organization still needs to manage application

deployments to those resources and manage those deployments to respond to the needs of the

business over time Container orchestration simplifies the solution to this problem the

containerization of the application makes the application easy to deploy in a variety of

6 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

environments and the container orchestrator provides a way to manage and scale that application

as needed

Container Orchestration Features

Kubernetes provides several features to accomplish the task of orchestrating containerized

applications

Declarative Management

Kubernetes is designed to be managed in a declarative manner rather than an imperative manner

An example of imperative management is installing a program via the apt-get install

command which imperatively installs the application in a certain location When an update to the

program is needed you imperatively tell apt-get to update the application

Declarative management is managing based on desired state Deployments are a Kubernetes

component designed to hold the state of an application Defined in JSON or YAML the

Deployment contains information about the container image such as its version the number of

containers that should exist within the Kubernetes cluster and a variety of other properties that

Kubernetes needs in order to deploy the application properly You simply state the number of

application instances that you want to have and Kubernetes creates and maintains the necessary

number of containers across your resources If a container fails for any reason (for example the

virtual machine on which the container is running goes offline) Kubernetes automatically stands

up a new one on a healthy node to maintain the state declared in the Deployment Kubernetes

constantly checks the actual state against the desired state and ensures that the two states match

The declarative model provides a benefit over the imperative model in which the system must

always be told what to do In the declarative model Kubernetes does the work for you as long as

the definitions for the desired system state exists

7 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Rolling Upgrades and Rollbacks

Another benefit of deploying applications via Kubernetes Deployments is its built-in rolling upgrade

and rollback functionality For example when a YAML file that defines the state of a Deployment is

updated with a new version of the container image Kubernetes recognizes this change and begins

to shut down existing instances of the older version while creating new instances with the updated

version While Kubernetes performs this rolling upgrade it continues to direct requests to running

container instances which normally results in zero downtime Likewise if there is an issue with the

upgrade Kubernetes performs a rollback as needed

Load Balancer

To manage connections to the applications deployed as containers within Kubernetes the

Kubernetes Service component provides a software-defined load balancer within the Kubernetes

cluster For example a deployment of three instances of the Java application might be accessed

by means of a single point in the Kubernetes Service As a result if an instance becomes

unavailable or if a rolling upgrade is performed the application can still be accessed without

interruption

These basic components of Kubernetes make it an excellent way to orchestrate large numbers of

containerized applications across a pool of resources As you consider how best to use container

orchestration to meet your needs learn more about Kubernetes by reading their documentation at

httpskubernetesiodocs

Cluster Architecture for Kubernetes Manual Deployment on Oracle Cloud Infrastructure

This section describes the components that make up a functioning Kubernetes cluster and

explains how those components are deployed across the compute resources that make up the

cluster The architecture described in this section will be deployed in the ldquoKubernetes Manual

Deployment Guide for Oracle Cloud Infrastructurerdquo (the next section)

A Kubernetes cluster consists of three main components etcd Kubernetes masters (or

controllers) and Kubernetes workers (or nodes) This guide explains how to create a highly

available (HA) Kubernetes cluster with the following architecture

Three etcd nodes (across three availability domains in one region)

Three Kubernetes masters (or controllers) (across three availability domains in one

region)

Three Kubernetes workers (or nodes) (across three availability domains in one region)

8 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The steps in this guide produce an infrastructure similar to the one shown in the following diagram

The etcd cluster is configured to run on a separate set of compute resources from the Kubernetes

cluster

The following sections explain the components in greater detail

etcd

etcd is a key-value store created by CoreOS Kubernetesrsquo state information is stored in the etcd

cluster This should not be confused with running an etcd cluster via Kubernetes rather this etcd

cluster is helping to run Kubernetes itself

In this guide the etcd cluster is configured to run on a separate set of compute resources from the

Kubernetes cluster Running etcd on separate compute resources provides greater isolation

between etcd and the components of the Kubernetes cluster

Kubernetes Masters

The Kubernetes masters (or controllers) are machines (virtual or physical) that run the API server

controller manager and scheduler components of the Kubernetes cluster

9 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Kubernetes Workers

The Kubernetes workers (or nodes) are machines (virtual or physical) that run the kubelet

component of the Kubernetes cluster The workers are the resources on which Kubernetes

schedules containers (or pods)

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure

This guide explains how to deploy and configure all the components and features required to run

Kubernetes on Oracle Cloud Infrastructure This guide assumes that you are starting with a ldquoclean

slaterdquo environment during the course of this guide you will create all the resources that you need

in Oracle Cloud Infrastructure

This guide walks through the following tasks

1 Create Oracle Cloud Infrastructure resources

2 Generate certificates for Kubernetes components

3 Bootstrap an HA etcd cluster

4 Generate token and configuration files to authenticate components

5 Bootstrap the HA Kubernetes masters (Kubernetes control plane)

6 Add a worker

7 Configure remote access

8 Deploy Kube-DNS

9 Smoke test

Step 1 Create Oracle Cloud Infrastructure Resources

This guide does not explain how to create the Oracle Cloud Infrastructure resources that you need

to create the Kubernetes cluster This section lists the required resources but you must create

them on your own by using the Oracle Cloud Infrastructure Console CLI API SDKs or the

Terraform provider For instructions see the Oracle Cloud Infrastructure documentation

Networking

To create the Kubernetes cluster in this guide you need the following Networking components

For more information about creating Networking components see the Networking section of the

Oracle Cloud Infrastructure documentation

10 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

VCN

You need a single virtual cloud network (VCN) with the following values

VCN Name CIDR Description

k8sOCIoraclevcncom 1000016 VCN used to host network resources for the

Kubernetes cluster

Subnets

You need a VCN and at least one subnet per availability domain (three subnets total) In a

production configuration we recommend creating an etcd master and worker subnet per

availability domain which would result in nine subnets For a test or learning deployment intended

to be removed later creating three subnets (one per availability domain) is sufficient The following

values describe the recommended configuration for a cluster in a region with three availability

domains

Use the following values

Subnet Name CIDR Availability

Domain Description

publicETCDSubnetAD1sub 10020024 AD1 Subnet used for etcd host in AD1

publicETCDSubnetAD2sub 10021024 AD2 Subnet used for etcd host in AD2

publicETCDSubnetAD3sub 10022024 AD3 Subnet used for etcd host in AD3

publicK8SMasterSubnetAD1sub 10030024 AD1 Subnet used for Kubernetes

masters in AD1

publicK8SMasterSubnetAD2sub 10031024 AD2 Subnet used for Kubernetes

masters in AD2

publicK8SMasterSubnetAD3sub 10032024 AD3 Subnet used for Kubernetes

masters in AD3

publicK8SWorkerSubnetAD1sub 10040024 AD1 Subnet used to host Kubernetes

workers in AD1

publicK8SWorkerSubnetAD2sub 10041024 AD2 Subnet used to host Kubernetes

workers in AD2

publicK8SWorkerSubnetAD3sub 10042024 AD3 Subnet used to host Kubernetes

workers in AD3

11 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Security Lists

A production configuration should include the following security lists

etcd_security_list

k8sMaster_security_list

k8sWorker_security_list

For a list of the recommended security rules for each security list see ldquoAppendix A Security

Rulesrdquo

Internet Gateway and Route Table

Your configuration should include one internet gateway and one route table rule that allows your

Kubernetes cluster to access the internet through the internet gateway

The route table rule should have a destination CIDR block of 00000 and target type of

internet gateway The target should be the internet gateway that you intend to use with your

Kubernetes cluster

Load Balancer

The recommended cluster configuration requires two load balancers Create a private load

balancer for your etcd nodes and a public load balancer for your Kubernetes masters Populate

the following table with your load balancerrsquos information to refer to throughout the guide

In the guide the public IP address of your Kubernetes master load balancer is referred to as

loadbalancer_public_ip

Use the following values

Load Balancer

Name

Load

Balancer

Type

Subnet1 Subnet2 Public IP Private IP

Compute

Resource

Back Ends

lb-etcd Private Not applicable Etcd1

Etcd2

Etcd3

lb-k8smaster Public KubeM1

KubeM2

KubeM3

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 4: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

4 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Target Audience

This document is intended for customers who are interested in learning how Kubernetes works by

deploying it on Oracle Cloud Infrastructure or who have considerable experience with Kubernetes

and want a baseline deployment process in order to create their own highly configured clusters

This document assumes that the user is working from a computer running a macOS operating

system and has an understanding of UNIX commands and file systems

Users of this document should also be familiar with the fundamentals of the Oracle Cloud

Infrastructure For information go to httpsdocsus-phoenix-1oraclecloudcom If this is the first

time that you have used the platform we recommend specifically the tutorial at httpsdocsus-

phoenix-1oraclecloudcomContentGSGReferenceoverviewworkflowhtm

Introduction

Kubernetes is the most popular container orchestration tool available today Although the

Kubernetes open-source project is still young and experiencing tremendous growth when it is

deployed properly Kubernetes can be a reliable tool for running container workloads in production

This document presents a starting point for deploying a secure highly available Kubernetes cluster

on Oracle Cloud Infrastructure The cluster created from the instructions in this document might be

sufficient for your needs However if you want to configure your cluster beyond what is presented

here yoursquoll need supplementary materials and white papers to address the various customization

options and updates to Kubernetes

Kubernetes is an incredibly fast-moving project with frequent new releases and bug fixes

Accordingly this document addresses deploying Kubernetes version 16 (the stable build at the

time of writing) Future documents will address the process to upgrade the cluster made in this

document

Overview of Kubernetes

This section provides a brief introduction to Kubernetes If you are already familiar with

Kubernetes you can skip this section

Kubernetes is the most popular container orchestration tool available and is maintained by one of

the fastest-growing open-source communities The Kubernetes project originated within Google a

long-time user of massive numbers of containers To manage these containers well they needed

to develop a system for container orchestration Kubernetes combines the lessons that Google

5 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

learned from years of container usage into a single tool with an array of features that make

container orchestration simple and adaptable to the wide variety of use cases in the technology

industry Since it became open source in July 2015 the capabilities of Kubernetes have continued

to grow Issues and new feature requests are tracked on the public GitHub project with new major

versions released approximately every two months

Containers are designed to solve problems with traditional application deployment such as

missing dependencies when installing an application or trouble deploying applications on specific

OS versions Container orchestrators aim to solve problems with scaling these applications

Container Images

The application with all of its dependencies is kept in a container image When run by a container

engine such as Docker the container image runs as a container The process of creating this

container image for an application is known as containerization Containerization is beneficial in

cases where the application would interact poorly with other applications if deployed on the same

machine The container provides a level of isolation that although not fully multi-tenant can

prevent applications from causing problems with other applications running on the same physical

host For example containers simplify Java application deployment by bundling the applicationrsquos

dependencies (like the specific Java Runtime Environment version) with the application in the

container Additionally if the application runs on Linux the container also abstracts the flavor and

version of the Linux OS All dependencies required to run the application residing in the OS can

also be bundled in the container As a result a containerized application runs the same on Oracle

Linux as it would on Ubuntu

Application Deployment

After the application and its dependencies are bundled in a container image the next step is to

distribute that application to all the groups that need it Assume that many teams need to use this

application and that the scale of the teamsrsquo usage might change over time In a traditional data

center scaling this application would likely require an analysis to estimate the resources needed to

run this application across the company for the next quarter or perhaps the entire next fiscal year

The IT organization that manages these physical resources would need to order new equipment to

satisfy the needs of the business In the cloud new resources on which to run the application can

be acquired on-demand and in a greater variety of sizes (that is a virtual machine with fewer cores

rather than a whole physical machine) however the organization still needs to manage application

deployments to those resources and manage those deployments to respond to the needs of the

business over time Container orchestration simplifies the solution to this problem the

containerization of the application makes the application easy to deploy in a variety of

6 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

environments and the container orchestrator provides a way to manage and scale that application

as needed

Container Orchestration Features

Kubernetes provides several features to accomplish the task of orchestrating containerized

applications

Declarative Management

Kubernetes is designed to be managed in a declarative manner rather than an imperative manner

An example of imperative management is installing a program via the apt-get install

command which imperatively installs the application in a certain location When an update to the

program is needed you imperatively tell apt-get to update the application

Declarative management is managing based on desired state Deployments are a Kubernetes

component designed to hold the state of an application Defined in JSON or YAML the

Deployment contains information about the container image such as its version the number of

containers that should exist within the Kubernetes cluster and a variety of other properties that

Kubernetes needs in order to deploy the application properly You simply state the number of

application instances that you want to have and Kubernetes creates and maintains the necessary

number of containers across your resources If a container fails for any reason (for example the

virtual machine on which the container is running goes offline) Kubernetes automatically stands

up a new one on a healthy node to maintain the state declared in the Deployment Kubernetes

constantly checks the actual state against the desired state and ensures that the two states match

The declarative model provides a benefit over the imperative model in which the system must

always be told what to do In the declarative model Kubernetes does the work for you as long as

the definitions for the desired system state exists

7 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Rolling Upgrades and Rollbacks

Another benefit of deploying applications via Kubernetes Deployments is its built-in rolling upgrade

and rollback functionality For example when a YAML file that defines the state of a Deployment is

updated with a new version of the container image Kubernetes recognizes this change and begins

to shut down existing instances of the older version while creating new instances with the updated

version While Kubernetes performs this rolling upgrade it continues to direct requests to running

container instances which normally results in zero downtime Likewise if there is an issue with the

upgrade Kubernetes performs a rollback as needed

Load Balancer

To manage connections to the applications deployed as containers within Kubernetes the

Kubernetes Service component provides a software-defined load balancer within the Kubernetes

cluster For example a deployment of three instances of the Java application might be accessed

by means of a single point in the Kubernetes Service As a result if an instance becomes

unavailable or if a rolling upgrade is performed the application can still be accessed without

interruption

These basic components of Kubernetes make it an excellent way to orchestrate large numbers of

containerized applications across a pool of resources As you consider how best to use container

orchestration to meet your needs learn more about Kubernetes by reading their documentation at

httpskubernetesiodocs

Cluster Architecture for Kubernetes Manual Deployment on Oracle Cloud Infrastructure

This section describes the components that make up a functioning Kubernetes cluster and

explains how those components are deployed across the compute resources that make up the

cluster The architecture described in this section will be deployed in the ldquoKubernetes Manual

Deployment Guide for Oracle Cloud Infrastructurerdquo (the next section)

A Kubernetes cluster consists of three main components etcd Kubernetes masters (or

controllers) and Kubernetes workers (or nodes) This guide explains how to create a highly

available (HA) Kubernetes cluster with the following architecture

Three etcd nodes (across three availability domains in one region)

Three Kubernetes masters (or controllers) (across three availability domains in one

region)

Three Kubernetes workers (or nodes) (across three availability domains in one region)

8 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The steps in this guide produce an infrastructure similar to the one shown in the following diagram

The etcd cluster is configured to run on a separate set of compute resources from the Kubernetes

cluster

The following sections explain the components in greater detail

etcd

etcd is a key-value store created by CoreOS Kubernetesrsquo state information is stored in the etcd

cluster This should not be confused with running an etcd cluster via Kubernetes rather this etcd

cluster is helping to run Kubernetes itself

In this guide the etcd cluster is configured to run on a separate set of compute resources from the

Kubernetes cluster Running etcd on separate compute resources provides greater isolation

between etcd and the components of the Kubernetes cluster

Kubernetes Masters

The Kubernetes masters (or controllers) are machines (virtual or physical) that run the API server

controller manager and scheduler components of the Kubernetes cluster

9 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Kubernetes Workers

The Kubernetes workers (or nodes) are machines (virtual or physical) that run the kubelet

component of the Kubernetes cluster The workers are the resources on which Kubernetes

schedules containers (or pods)

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure

This guide explains how to deploy and configure all the components and features required to run

Kubernetes on Oracle Cloud Infrastructure This guide assumes that you are starting with a ldquoclean

slaterdquo environment during the course of this guide you will create all the resources that you need

in Oracle Cloud Infrastructure

This guide walks through the following tasks

1 Create Oracle Cloud Infrastructure resources

2 Generate certificates for Kubernetes components

3 Bootstrap an HA etcd cluster

4 Generate token and configuration files to authenticate components

5 Bootstrap the HA Kubernetes masters (Kubernetes control plane)

6 Add a worker

7 Configure remote access

8 Deploy Kube-DNS

9 Smoke test

Step 1 Create Oracle Cloud Infrastructure Resources

This guide does not explain how to create the Oracle Cloud Infrastructure resources that you need

to create the Kubernetes cluster This section lists the required resources but you must create

them on your own by using the Oracle Cloud Infrastructure Console CLI API SDKs or the

Terraform provider For instructions see the Oracle Cloud Infrastructure documentation

Networking

To create the Kubernetes cluster in this guide you need the following Networking components

For more information about creating Networking components see the Networking section of the

Oracle Cloud Infrastructure documentation

10 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

VCN

You need a single virtual cloud network (VCN) with the following values

VCN Name CIDR Description

k8sOCIoraclevcncom 1000016 VCN used to host network resources for the

Kubernetes cluster

Subnets

You need a VCN and at least one subnet per availability domain (three subnets total) In a

production configuration we recommend creating an etcd master and worker subnet per

availability domain which would result in nine subnets For a test or learning deployment intended

to be removed later creating three subnets (one per availability domain) is sufficient The following

values describe the recommended configuration for a cluster in a region with three availability

domains

Use the following values

Subnet Name CIDR Availability

Domain Description

publicETCDSubnetAD1sub 10020024 AD1 Subnet used for etcd host in AD1

publicETCDSubnetAD2sub 10021024 AD2 Subnet used for etcd host in AD2

publicETCDSubnetAD3sub 10022024 AD3 Subnet used for etcd host in AD3

publicK8SMasterSubnetAD1sub 10030024 AD1 Subnet used for Kubernetes

masters in AD1

publicK8SMasterSubnetAD2sub 10031024 AD2 Subnet used for Kubernetes

masters in AD2

publicK8SMasterSubnetAD3sub 10032024 AD3 Subnet used for Kubernetes

masters in AD3

publicK8SWorkerSubnetAD1sub 10040024 AD1 Subnet used to host Kubernetes

workers in AD1

publicK8SWorkerSubnetAD2sub 10041024 AD2 Subnet used to host Kubernetes

workers in AD2

publicK8SWorkerSubnetAD3sub 10042024 AD3 Subnet used to host Kubernetes

workers in AD3

11 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Security Lists

A production configuration should include the following security lists

etcd_security_list

k8sMaster_security_list

k8sWorker_security_list

For a list of the recommended security rules for each security list see ldquoAppendix A Security

Rulesrdquo

Internet Gateway and Route Table

Your configuration should include one internet gateway and one route table rule that allows your

Kubernetes cluster to access the internet through the internet gateway

The route table rule should have a destination CIDR block of 00000 and target type of

internet gateway The target should be the internet gateway that you intend to use with your

Kubernetes cluster

Load Balancer

The recommended cluster configuration requires two load balancers Create a private load

balancer for your etcd nodes and a public load balancer for your Kubernetes masters Populate

the following table with your load balancerrsquos information to refer to throughout the guide

In the guide the public IP address of your Kubernetes master load balancer is referred to as

loadbalancer_public_ip

Use the following values

Load Balancer

Name

Load

Balancer

Type

Subnet1 Subnet2 Public IP Private IP

Compute

Resource

Back Ends

lb-etcd Private Not applicable Etcd1

Etcd2

Etcd3

lb-k8smaster Public KubeM1

KubeM2

KubeM3

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 5: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

5 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

learned from years of container usage into a single tool with an array of features that make

container orchestration simple and adaptable to the wide variety of use cases in the technology

industry Since it became open source in July 2015 the capabilities of Kubernetes have continued

to grow Issues and new feature requests are tracked on the public GitHub project with new major

versions released approximately every two months

Containers are designed to solve problems with traditional application deployment such as

missing dependencies when installing an application or trouble deploying applications on specific

OS versions Container orchestrators aim to solve problems with scaling these applications

Container Images

The application with all of its dependencies is kept in a container image When run by a container

engine such as Docker the container image runs as a container The process of creating this

container image for an application is known as containerization Containerization is beneficial in

cases where the application would interact poorly with other applications if deployed on the same

machine The container provides a level of isolation that although not fully multi-tenant can

prevent applications from causing problems with other applications running on the same physical

host For example containers simplify Java application deployment by bundling the applicationrsquos

dependencies (like the specific Java Runtime Environment version) with the application in the

container Additionally if the application runs on Linux the container also abstracts the flavor and

version of the Linux OS All dependencies required to run the application residing in the OS can

also be bundled in the container As a result a containerized application runs the same on Oracle

Linux as it would on Ubuntu

Application Deployment

After the application and its dependencies are bundled in a container image the next step is to

distribute that application to all the groups that need it Assume that many teams need to use this

application and that the scale of the teamsrsquo usage might change over time In a traditional data

center scaling this application would likely require an analysis to estimate the resources needed to

run this application across the company for the next quarter or perhaps the entire next fiscal year

The IT organization that manages these physical resources would need to order new equipment to

satisfy the needs of the business In the cloud new resources on which to run the application can

be acquired on-demand and in a greater variety of sizes (that is a virtual machine with fewer cores

rather than a whole physical machine) however the organization still needs to manage application

deployments to those resources and manage those deployments to respond to the needs of the

business over time Container orchestration simplifies the solution to this problem the

containerization of the application makes the application easy to deploy in a variety of

6 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

environments and the container orchestrator provides a way to manage and scale that application

as needed

Container Orchestration Features

Kubernetes provides several features to accomplish the task of orchestrating containerized

applications

Declarative Management

Kubernetes is designed to be managed in a declarative manner rather than an imperative manner

An example of imperative management is installing a program via the apt-get install

command which imperatively installs the application in a certain location When an update to the

program is needed you imperatively tell apt-get to update the application

Declarative management is managing based on desired state Deployments are a Kubernetes

component designed to hold the state of an application Defined in JSON or YAML the

Deployment contains information about the container image such as its version the number of

containers that should exist within the Kubernetes cluster and a variety of other properties that

Kubernetes needs in order to deploy the application properly You simply state the number of

application instances that you want to have and Kubernetes creates and maintains the necessary

number of containers across your resources If a container fails for any reason (for example the

virtual machine on which the container is running goes offline) Kubernetes automatically stands

up a new one on a healthy node to maintain the state declared in the Deployment Kubernetes

constantly checks the actual state against the desired state and ensures that the two states match

The declarative model provides a benefit over the imperative model in which the system must

always be told what to do In the declarative model Kubernetes does the work for you as long as

the definitions for the desired system state exists

7 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Rolling Upgrades and Rollbacks

Another benefit of deploying applications via Kubernetes Deployments is its built-in rolling upgrade

and rollback functionality For example when a YAML file that defines the state of a Deployment is

updated with a new version of the container image Kubernetes recognizes this change and begins

to shut down existing instances of the older version while creating new instances with the updated

version While Kubernetes performs this rolling upgrade it continues to direct requests to running

container instances which normally results in zero downtime Likewise if there is an issue with the

upgrade Kubernetes performs a rollback as needed

Load Balancer

To manage connections to the applications deployed as containers within Kubernetes the

Kubernetes Service component provides a software-defined load balancer within the Kubernetes

cluster For example a deployment of three instances of the Java application might be accessed

by means of a single point in the Kubernetes Service As a result if an instance becomes

unavailable or if a rolling upgrade is performed the application can still be accessed without

interruption

These basic components of Kubernetes make it an excellent way to orchestrate large numbers of

containerized applications across a pool of resources As you consider how best to use container

orchestration to meet your needs learn more about Kubernetes by reading their documentation at

httpskubernetesiodocs

Cluster Architecture for Kubernetes Manual Deployment on Oracle Cloud Infrastructure

This section describes the components that make up a functioning Kubernetes cluster and

explains how those components are deployed across the compute resources that make up the

cluster The architecture described in this section will be deployed in the ldquoKubernetes Manual

Deployment Guide for Oracle Cloud Infrastructurerdquo (the next section)

A Kubernetes cluster consists of three main components etcd Kubernetes masters (or

controllers) and Kubernetes workers (or nodes) This guide explains how to create a highly

available (HA) Kubernetes cluster with the following architecture

Three etcd nodes (across three availability domains in one region)

Three Kubernetes masters (or controllers) (across three availability domains in one

region)

Three Kubernetes workers (or nodes) (across three availability domains in one region)

8 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The steps in this guide produce an infrastructure similar to the one shown in the following diagram

The etcd cluster is configured to run on a separate set of compute resources from the Kubernetes

cluster

The following sections explain the components in greater detail

etcd

etcd is a key-value store created by CoreOS Kubernetesrsquo state information is stored in the etcd

cluster This should not be confused with running an etcd cluster via Kubernetes rather this etcd

cluster is helping to run Kubernetes itself

In this guide the etcd cluster is configured to run on a separate set of compute resources from the

Kubernetes cluster Running etcd on separate compute resources provides greater isolation

between etcd and the components of the Kubernetes cluster

Kubernetes Masters

The Kubernetes masters (or controllers) are machines (virtual or physical) that run the API server

controller manager and scheduler components of the Kubernetes cluster

9 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Kubernetes Workers

The Kubernetes workers (or nodes) are machines (virtual or physical) that run the kubelet

component of the Kubernetes cluster The workers are the resources on which Kubernetes

schedules containers (or pods)

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure

This guide explains how to deploy and configure all the components and features required to run

Kubernetes on Oracle Cloud Infrastructure This guide assumes that you are starting with a ldquoclean

slaterdquo environment during the course of this guide you will create all the resources that you need

in Oracle Cloud Infrastructure

This guide walks through the following tasks

1 Create Oracle Cloud Infrastructure resources

2 Generate certificates for Kubernetes components

3 Bootstrap an HA etcd cluster

4 Generate token and configuration files to authenticate components

5 Bootstrap the HA Kubernetes masters (Kubernetes control plane)

6 Add a worker

7 Configure remote access

8 Deploy Kube-DNS

9 Smoke test

Step 1 Create Oracle Cloud Infrastructure Resources

This guide does not explain how to create the Oracle Cloud Infrastructure resources that you need

to create the Kubernetes cluster This section lists the required resources but you must create

them on your own by using the Oracle Cloud Infrastructure Console CLI API SDKs or the

Terraform provider For instructions see the Oracle Cloud Infrastructure documentation

Networking

To create the Kubernetes cluster in this guide you need the following Networking components

For more information about creating Networking components see the Networking section of the

Oracle Cloud Infrastructure documentation

10 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

VCN

You need a single virtual cloud network (VCN) with the following values

VCN Name CIDR Description

k8sOCIoraclevcncom 1000016 VCN used to host network resources for the

Kubernetes cluster

Subnets

You need a VCN and at least one subnet per availability domain (three subnets total) In a

production configuration we recommend creating an etcd master and worker subnet per

availability domain which would result in nine subnets For a test or learning deployment intended

to be removed later creating three subnets (one per availability domain) is sufficient The following

values describe the recommended configuration for a cluster in a region with three availability

domains

Use the following values

Subnet Name CIDR Availability

Domain Description

publicETCDSubnetAD1sub 10020024 AD1 Subnet used for etcd host in AD1

publicETCDSubnetAD2sub 10021024 AD2 Subnet used for etcd host in AD2

publicETCDSubnetAD3sub 10022024 AD3 Subnet used for etcd host in AD3

publicK8SMasterSubnetAD1sub 10030024 AD1 Subnet used for Kubernetes

masters in AD1

publicK8SMasterSubnetAD2sub 10031024 AD2 Subnet used for Kubernetes

masters in AD2

publicK8SMasterSubnetAD3sub 10032024 AD3 Subnet used for Kubernetes

masters in AD3

publicK8SWorkerSubnetAD1sub 10040024 AD1 Subnet used to host Kubernetes

workers in AD1

publicK8SWorkerSubnetAD2sub 10041024 AD2 Subnet used to host Kubernetes

workers in AD2

publicK8SWorkerSubnetAD3sub 10042024 AD3 Subnet used to host Kubernetes

workers in AD3

11 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Security Lists

A production configuration should include the following security lists

etcd_security_list

k8sMaster_security_list

k8sWorker_security_list

For a list of the recommended security rules for each security list see ldquoAppendix A Security

Rulesrdquo

Internet Gateway and Route Table

Your configuration should include one internet gateway and one route table rule that allows your

Kubernetes cluster to access the internet through the internet gateway

The route table rule should have a destination CIDR block of 00000 and target type of

internet gateway The target should be the internet gateway that you intend to use with your

Kubernetes cluster

Load Balancer

The recommended cluster configuration requires two load balancers Create a private load

balancer for your etcd nodes and a public load balancer for your Kubernetes masters Populate

the following table with your load balancerrsquos information to refer to throughout the guide

In the guide the public IP address of your Kubernetes master load balancer is referred to as

loadbalancer_public_ip

Use the following values

Load Balancer

Name

Load

Balancer

Type

Subnet1 Subnet2 Public IP Private IP

Compute

Resource

Back Ends

lb-etcd Private Not applicable Etcd1

Etcd2

Etcd3

lb-k8smaster Public KubeM1

KubeM2

KubeM3

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 6: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

6 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

environments and the container orchestrator provides a way to manage and scale that application

as needed

Container Orchestration Features

Kubernetes provides several features to accomplish the task of orchestrating containerized

applications

Declarative Management

Kubernetes is designed to be managed in a declarative manner rather than an imperative manner

An example of imperative management is installing a program via the apt-get install

command which imperatively installs the application in a certain location When an update to the

program is needed you imperatively tell apt-get to update the application

Declarative management is managing based on desired state Deployments are a Kubernetes

component designed to hold the state of an application Defined in JSON or YAML the

Deployment contains information about the container image such as its version the number of

containers that should exist within the Kubernetes cluster and a variety of other properties that

Kubernetes needs in order to deploy the application properly You simply state the number of

application instances that you want to have and Kubernetes creates and maintains the necessary

number of containers across your resources If a container fails for any reason (for example the

virtual machine on which the container is running goes offline) Kubernetes automatically stands

up a new one on a healthy node to maintain the state declared in the Deployment Kubernetes

constantly checks the actual state against the desired state and ensures that the two states match

The declarative model provides a benefit over the imperative model in which the system must

always be told what to do In the declarative model Kubernetes does the work for you as long as

the definitions for the desired system state exists

7 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Rolling Upgrades and Rollbacks

Another benefit of deploying applications via Kubernetes Deployments is its built-in rolling upgrade

and rollback functionality For example when a YAML file that defines the state of a Deployment is

updated with a new version of the container image Kubernetes recognizes this change and begins

to shut down existing instances of the older version while creating new instances with the updated

version While Kubernetes performs this rolling upgrade it continues to direct requests to running

container instances which normally results in zero downtime Likewise if there is an issue with the

upgrade Kubernetes performs a rollback as needed

Load Balancer

To manage connections to the applications deployed as containers within Kubernetes the

Kubernetes Service component provides a software-defined load balancer within the Kubernetes

cluster For example a deployment of three instances of the Java application might be accessed

by means of a single point in the Kubernetes Service As a result if an instance becomes

unavailable or if a rolling upgrade is performed the application can still be accessed without

interruption

These basic components of Kubernetes make it an excellent way to orchestrate large numbers of

containerized applications across a pool of resources As you consider how best to use container

orchestration to meet your needs learn more about Kubernetes by reading their documentation at

httpskubernetesiodocs

Cluster Architecture for Kubernetes Manual Deployment on Oracle Cloud Infrastructure

This section describes the components that make up a functioning Kubernetes cluster and

explains how those components are deployed across the compute resources that make up the

cluster The architecture described in this section will be deployed in the ldquoKubernetes Manual

Deployment Guide for Oracle Cloud Infrastructurerdquo (the next section)

A Kubernetes cluster consists of three main components etcd Kubernetes masters (or

controllers) and Kubernetes workers (or nodes) This guide explains how to create a highly

available (HA) Kubernetes cluster with the following architecture

Three etcd nodes (across three availability domains in one region)

Three Kubernetes masters (or controllers) (across three availability domains in one

region)

Three Kubernetes workers (or nodes) (across three availability domains in one region)

8 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The steps in this guide produce an infrastructure similar to the one shown in the following diagram

The etcd cluster is configured to run on a separate set of compute resources from the Kubernetes

cluster

The following sections explain the components in greater detail

etcd

etcd is a key-value store created by CoreOS Kubernetesrsquo state information is stored in the etcd

cluster This should not be confused with running an etcd cluster via Kubernetes rather this etcd

cluster is helping to run Kubernetes itself

In this guide the etcd cluster is configured to run on a separate set of compute resources from the

Kubernetes cluster Running etcd on separate compute resources provides greater isolation

between etcd and the components of the Kubernetes cluster

Kubernetes Masters

The Kubernetes masters (or controllers) are machines (virtual or physical) that run the API server

controller manager and scheduler components of the Kubernetes cluster

9 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Kubernetes Workers

The Kubernetes workers (or nodes) are machines (virtual or physical) that run the kubelet

component of the Kubernetes cluster The workers are the resources on which Kubernetes

schedules containers (or pods)

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure

This guide explains how to deploy and configure all the components and features required to run

Kubernetes on Oracle Cloud Infrastructure This guide assumes that you are starting with a ldquoclean

slaterdquo environment during the course of this guide you will create all the resources that you need

in Oracle Cloud Infrastructure

This guide walks through the following tasks

1 Create Oracle Cloud Infrastructure resources

2 Generate certificates for Kubernetes components

3 Bootstrap an HA etcd cluster

4 Generate token and configuration files to authenticate components

5 Bootstrap the HA Kubernetes masters (Kubernetes control plane)

6 Add a worker

7 Configure remote access

8 Deploy Kube-DNS

9 Smoke test

Step 1 Create Oracle Cloud Infrastructure Resources

This guide does not explain how to create the Oracle Cloud Infrastructure resources that you need

to create the Kubernetes cluster This section lists the required resources but you must create

them on your own by using the Oracle Cloud Infrastructure Console CLI API SDKs or the

Terraform provider For instructions see the Oracle Cloud Infrastructure documentation

Networking

To create the Kubernetes cluster in this guide you need the following Networking components

For more information about creating Networking components see the Networking section of the

Oracle Cloud Infrastructure documentation

10 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

VCN

You need a single virtual cloud network (VCN) with the following values

VCN Name CIDR Description

k8sOCIoraclevcncom 1000016 VCN used to host network resources for the

Kubernetes cluster

Subnets

You need a VCN and at least one subnet per availability domain (three subnets total) In a

production configuration we recommend creating an etcd master and worker subnet per

availability domain which would result in nine subnets For a test or learning deployment intended

to be removed later creating three subnets (one per availability domain) is sufficient The following

values describe the recommended configuration for a cluster in a region with three availability

domains

Use the following values

Subnet Name CIDR Availability

Domain Description

publicETCDSubnetAD1sub 10020024 AD1 Subnet used for etcd host in AD1

publicETCDSubnetAD2sub 10021024 AD2 Subnet used for etcd host in AD2

publicETCDSubnetAD3sub 10022024 AD3 Subnet used for etcd host in AD3

publicK8SMasterSubnetAD1sub 10030024 AD1 Subnet used for Kubernetes

masters in AD1

publicK8SMasterSubnetAD2sub 10031024 AD2 Subnet used for Kubernetes

masters in AD2

publicK8SMasterSubnetAD3sub 10032024 AD3 Subnet used for Kubernetes

masters in AD3

publicK8SWorkerSubnetAD1sub 10040024 AD1 Subnet used to host Kubernetes

workers in AD1

publicK8SWorkerSubnetAD2sub 10041024 AD2 Subnet used to host Kubernetes

workers in AD2

publicK8SWorkerSubnetAD3sub 10042024 AD3 Subnet used to host Kubernetes

workers in AD3

11 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Security Lists

A production configuration should include the following security lists

etcd_security_list

k8sMaster_security_list

k8sWorker_security_list

For a list of the recommended security rules for each security list see ldquoAppendix A Security

Rulesrdquo

Internet Gateway and Route Table

Your configuration should include one internet gateway and one route table rule that allows your

Kubernetes cluster to access the internet through the internet gateway

The route table rule should have a destination CIDR block of 00000 and target type of

internet gateway The target should be the internet gateway that you intend to use with your

Kubernetes cluster

Load Balancer

The recommended cluster configuration requires two load balancers Create a private load

balancer for your etcd nodes and a public load balancer for your Kubernetes masters Populate

the following table with your load balancerrsquos information to refer to throughout the guide

In the guide the public IP address of your Kubernetes master load balancer is referred to as

loadbalancer_public_ip

Use the following values

Load Balancer

Name

Load

Balancer

Type

Subnet1 Subnet2 Public IP Private IP

Compute

Resource

Back Ends

lb-etcd Private Not applicable Etcd1

Etcd2

Etcd3

lb-k8smaster Public KubeM1

KubeM2

KubeM3

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 7: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

7 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Rolling Upgrades and Rollbacks

Another benefit of deploying applications via Kubernetes Deployments is its built-in rolling upgrade

and rollback functionality For example when a YAML file that defines the state of a Deployment is

updated with a new version of the container image Kubernetes recognizes this change and begins

to shut down existing instances of the older version while creating new instances with the updated

version While Kubernetes performs this rolling upgrade it continues to direct requests to running

container instances which normally results in zero downtime Likewise if there is an issue with the

upgrade Kubernetes performs a rollback as needed

Load Balancer

To manage connections to the applications deployed as containers within Kubernetes the

Kubernetes Service component provides a software-defined load balancer within the Kubernetes

cluster For example a deployment of three instances of the Java application might be accessed

by means of a single point in the Kubernetes Service As a result if an instance becomes

unavailable or if a rolling upgrade is performed the application can still be accessed without

interruption

These basic components of Kubernetes make it an excellent way to orchestrate large numbers of

containerized applications across a pool of resources As you consider how best to use container

orchestration to meet your needs learn more about Kubernetes by reading their documentation at

httpskubernetesiodocs

Cluster Architecture for Kubernetes Manual Deployment on Oracle Cloud Infrastructure

This section describes the components that make up a functioning Kubernetes cluster and

explains how those components are deployed across the compute resources that make up the

cluster The architecture described in this section will be deployed in the ldquoKubernetes Manual

Deployment Guide for Oracle Cloud Infrastructurerdquo (the next section)

A Kubernetes cluster consists of three main components etcd Kubernetes masters (or

controllers) and Kubernetes workers (or nodes) This guide explains how to create a highly

available (HA) Kubernetes cluster with the following architecture

Three etcd nodes (across three availability domains in one region)

Three Kubernetes masters (or controllers) (across three availability domains in one

region)

Three Kubernetes workers (or nodes) (across three availability domains in one region)

8 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The steps in this guide produce an infrastructure similar to the one shown in the following diagram

The etcd cluster is configured to run on a separate set of compute resources from the Kubernetes

cluster

The following sections explain the components in greater detail

etcd

etcd is a key-value store created by CoreOS Kubernetesrsquo state information is stored in the etcd

cluster This should not be confused with running an etcd cluster via Kubernetes rather this etcd

cluster is helping to run Kubernetes itself

In this guide the etcd cluster is configured to run on a separate set of compute resources from the

Kubernetes cluster Running etcd on separate compute resources provides greater isolation

between etcd and the components of the Kubernetes cluster

Kubernetes Masters

The Kubernetes masters (or controllers) are machines (virtual or physical) that run the API server

controller manager and scheduler components of the Kubernetes cluster

9 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Kubernetes Workers

The Kubernetes workers (or nodes) are machines (virtual or physical) that run the kubelet

component of the Kubernetes cluster The workers are the resources on which Kubernetes

schedules containers (or pods)

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure

This guide explains how to deploy and configure all the components and features required to run

Kubernetes on Oracle Cloud Infrastructure This guide assumes that you are starting with a ldquoclean

slaterdquo environment during the course of this guide you will create all the resources that you need

in Oracle Cloud Infrastructure

This guide walks through the following tasks

1 Create Oracle Cloud Infrastructure resources

2 Generate certificates for Kubernetes components

3 Bootstrap an HA etcd cluster

4 Generate token and configuration files to authenticate components

5 Bootstrap the HA Kubernetes masters (Kubernetes control plane)

6 Add a worker

7 Configure remote access

8 Deploy Kube-DNS

9 Smoke test

Step 1 Create Oracle Cloud Infrastructure Resources

This guide does not explain how to create the Oracle Cloud Infrastructure resources that you need

to create the Kubernetes cluster This section lists the required resources but you must create

them on your own by using the Oracle Cloud Infrastructure Console CLI API SDKs or the

Terraform provider For instructions see the Oracle Cloud Infrastructure documentation

Networking

To create the Kubernetes cluster in this guide you need the following Networking components

For more information about creating Networking components see the Networking section of the

Oracle Cloud Infrastructure documentation

10 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

VCN

You need a single virtual cloud network (VCN) with the following values

VCN Name CIDR Description

k8sOCIoraclevcncom 1000016 VCN used to host network resources for the

Kubernetes cluster

Subnets

You need a VCN and at least one subnet per availability domain (three subnets total) In a

production configuration we recommend creating an etcd master and worker subnet per

availability domain which would result in nine subnets For a test or learning deployment intended

to be removed later creating three subnets (one per availability domain) is sufficient The following

values describe the recommended configuration for a cluster in a region with three availability

domains

Use the following values

Subnet Name CIDR Availability

Domain Description

publicETCDSubnetAD1sub 10020024 AD1 Subnet used for etcd host in AD1

publicETCDSubnetAD2sub 10021024 AD2 Subnet used for etcd host in AD2

publicETCDSubnetAD3sub 10022024 AD3 Subnet used for etcd host in AD3

publicK8SMasterSubnetAD1sub 10030024 AD1 Subnet used for Kubernetes

masters in AD1

publicK8SMasterSubnetAD2sub 10031024 AD2 Subnet used for Kubernetes

masters in AD2

publicK8SMasterSubnetAD3sub 10032024 AD3 Subnet used for Kubernetes

masters in AD3

publicK8SWorkerSubnetAD1sub 10040024 AD1 Subnet used to host Kubernetes

workers in AD1

publicK8SWorkerSubnetAD2sub 10041024 AD2 Subnet used to host Kubernetes

workers in AD2

publicK8SWorkerSubnetAD3sub 10042024 AD3 Subnet used to host Kubernetes

workers in AD3

11 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Security Lists

A production configuration should include the following security lists

etcd_security_list

k8sMaster_security_list

k8sWorker_security_list

For a list of the recommended security rules for each security list see ldquoAppendix A Security

Rulesrdquo

Internet Gateway and Route Table

Your configuration should include one internet gateway and one route table rule that allows your

Kubernetes cluster to access the internet through the internet gateway

The route table rule should have a destination CIDR block of 00000 and target type of

internet gateway The target should be the internet gateway that you intend to use with your

Kubernetes cluster

Load Balancer

The recommended cluster configuration requires two load balancers Create a private load

balancer for your etcd nodes and a public load balancer for your Kubernetes masters Populate

the following table with your load balancerrsquos information to refer to throughout the guide

In the guide the public IP address of your Kubernetes master load balancer is referred to as

loadbalancer_public_ip

Use the following values

Load Balancer

Name

Load

Balancer

Type

Subnet1 Subnet2 Public IP Private IP

Compute

Resource

Back Ends

lb-etcd Private Not applicable Etcd1

Etcd2

Etcd3

lb-k8smaster Public KubeM1

KubeM2

KubeM3

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 8: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

8 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The steps in this guide produce an infrastructure similar to the one shown in the following diagram

The etcd cluster is configured to run on a separate set of compute resources from the Kubernetes

cluster

The following sections explain the components in greater detail

etcd

etcd is a key-value store created by CoreOS Kubernetesrsquo state information is stored in the etcd

cluster This should not be confused with running an etcd cluster via Kubernetes rather this etcd

cluster is helping to run Kubernetes itself

In this guide the etcd cluster is configured to run on a separate set of compute resources from the

Kubernetes cluster Running etcd on separate compute resources provides greater isolation

between etcd and the components of the Kubernetes cluster

Kubernetes Masters

The Kubernetes masters (or controllers) are machines (virtual or physical) that run the API server

controller manager and scheduler components of the Kubernetes cluster

9 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Kubernetes Workers

The Kubernetes workers (or nodes) are machines (virtual or physical) that run the kubelet

component of the Kubernetes cluster The workers are the resources on which Kubernetes

schedules containers (or pods)

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure

This guide explains how to deploy and configure all the components and features required to run

Kubernetes on Oracle Cloud Infrastructure This guide assumes that you are starting with a ldquoclean

slaterdquo environment during the course of this guide you will create all the resources that you need

in Oracle Cloud Infrastructure

This guide walks through the following tasks

1 Create Oracle Cloud Infrastructure resources

2 Generate certificates for Kubernetes components

3 Bootstrap an HA etcd cluster

4 Generate token and configuration files to authenticate components

5 Bootstrap the HA Kubernetes masters (Kubernetes control plane)

6 Add a worker

7 Configure remote access

8 Deploy Kube-DNS

9 Smoke test

Step 1 Create Oracle Cloud Infrastructure Resources

This guide does not explain how to create the Oracle Cloud Infrastructure resources that you need

to create the Kubernetes cluster This section lists the required resources but you must create

them on your own by using the Oracle Cloud Infrastructure Console CLI API SDKs or the

Terraform provider For instructions see the Oracle Cloud Infrastructure documentation

Networking

To create the Kubernetes cluster in this guide you need the following Networking components

For more information about creating Networking components see the Networking section of the

Oracle Cloud Infrastructure documentation

10 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

VCN

You need a single virtual cloud network (VCN) with the following values

VCN Name CIDR Description

k8sOCIoraclevcncom 1000016 VCN used to host network resources for the

Kubernetes cluster

Subnets

You need a VCN and at least one subnet per availability domain (three subnets total) In a

production configuration we recommend creating an etcd master and worker subnet per

availability domain which would result in nine subnets For a test or learning deployment intended

to be removed later creating three subnets (one per availability domain) is sufficient The following

values describe the recommended configuration for a cluster in a region with three availability

domains

Use the following values

Subnet Name CIDR Availability

Domain Description

publicETCDSubnetAD1sub 10020024 AD1 Subnet used for etcd host in AD1

publicETCDSubnetAD2sub 10021024 AD2 Subnet used for etcd host in AD2

publicETCDSubnetAD3sub 10022024 AD3 Subnet used for etcd host in AD3

publicK8SMasterSubnetAD1sub 10030024 AD1 Subnet used for Kubernetes

masters in AD1

publicK8SMasterSubnetAD2sub 10031024 AD2 Subnet used for Kubernetes

masters in AD2

publicK8SMasterSubnetAD3sub 10032024 AD3 Subnet used for Kubernetes

masters in AD3

publicK8SWorkerSubnetAD1sub 10040024 AD1 Subnet used to host Kubernetes

workers in AD1

publicK8SWorkerSubnetAD2sub 10041024 AD2 Subnet used to host Kubernetes

workers in AD2

publicK8SWorkerSubnetAD3sub 10042024 AD3 Subnet used to host Kubernetes

workers in AD3

11 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Security Lists

A production configuration should include the following security lists

etcd_security_list

k8sMaster_security_list

k8sWorker_security_list

For a list of the recommended security rules for each security list see ldquoAppendix A Security

Rulesrdquo

Internet Gateway and Route Table

Your configuration should include one internet gateway and one route table rule that allows your

Kubernetes cluster to access the internet through the internet gateway

The route table rule should have a destination CIDR block of 00000 and target type of

internet gateway The target should be the internet gateway that you intend to use with your

Kubernetes cluster

Load Balancer

The recommended cluster configuration requires two load balancers Create a private load

balancer for your etcd nodes and a public load balancer for your Kubernetes masters Populate

the following table with your load balancerrsquos information to refer to throughout the guide

In the guide the public IP address of your Kubernetes master load balancer is referred to as

loadbalancer_public_ip

Use the following values

Load Balancer

Name

Load

Balancer

Type

Subnet1 Subnet2 Public IP Private IP

Compute

Resource

Back Ends

lb-etcd Private Not applicable Etcd1

Etcd2

Etcd3

lb-k8smaster Public KubeM1

KubeM2

KubeM3

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 9: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

9 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Kubernetes Workers

The Kubernetes workers (or nodes) are machines (virtual or physical) that run the kubelet

component of the Kubernetes cluster The workers are the resources on which Kubernetes

schedules containers (or pods)

Kubernetes Manual Deployment Guide for Oracle Cloud Infrastructure

This guide explains how to deploy and configure all the components and features required to run

Kubernetes on Oracle Cloud Infrastructure This guide assumes that you are starting with a ldquoclean

slaterdquo environment during the course of this guide you will create all the resources that you need

in Oracle Cloud Infrastructure

This guide walks through the following tasks

1 Create Oracle Cloud Infrastructure resources

2 Generate certificates for Kubernetes components

3 Bootstrap an HA etcd cluster

4 Generate token and configuration files to authenticate components

5 Bootstrap the HA Kubernetes masters (Kubernetes control plane)

6 Add a worker

7 Configure remote access

8 Deploy Kube-DNS

9 Smoke test

Step 1 Create Oracle Cloud Infrastructure Resources

This guide does not explain how to create the Oracle Cloud Infrastructure resources that you need

to create the Kubernetes cluster This section lists the required resources but you must create

them on your own by using the Oracle Cloud Infrastructure Console CLI API SDKs or the

Terraform provider For instructions see the Oracle Cloud Infrastructure documentation

Networking

To create the Kubernetes cluster in this guide you need the following Networking components

For more information about creating Networking components see the Networking section of the

Oracle Cloud Infrastructure documentation

10 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

VCN

You need a single virtual cloud network (VCN) with the following values

VCN Name CIDR Description

k8sOCIoraclevcncom 1000016 VCN used to host network resources for the

Kubernetes cluster

Subnets

You need a VCN and at least one subnet per availability domain (three subnets total) In a

production configuration we recommend creating an etcd master and worker subnet per

availability domain which would result in nine subnets For a test or learning deployment intended

to be removed later creating three subnets (one per availability domain) is sufficient The following

values describe the recommended configuration for a cluster in a region with three availability

domains

Use the following values

Subnet Name CIDR Availability

Domain Description

publicETCDSubnetAD1sub 10020024 AD1 Subnet used for etcd host in AD1

publicETCDSubnetAD2sub 10021024 AD2 Subnet used for etcd host in AD2

publicETCDSubnetAD3sub 10022024 AD3 Subnet used for etcd host in AD3

publicK8SMasterSubnetAD1sub 10030024 AD1 Subnet used for Kubernetes

masters in AD1

publicK8SMasterSubnetAD2sub 10031024 AD2 Subnet used for Kubernetes

masters in AD2

publicK8SMasterSubnetAD3sub 10032024 AD3 Subnet used for Kubernetes

masters in AD3

publicK8SWorkerSubnetAD1sub 10040024 AD1 Subnet used to host Kubernetes

workers in AD1

publicK8SWorkerSubnetAD2sub 10041024 AD2 Subnet used to host Kubernetes

workers in AD2

publicK8SWorkerSubnetAD3sub 10042024 AD3 Subnet used to host Kubernetes

workers in AD3

11 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Security Lists

A production configuration should include the following security lists

etcd_security_list

k8sMaster_security_list

k8sWorker_security_list

For a list of the recommended security rules for each security list see ldquoAppendix A Security

Rulesrdquo

Internet Gateway and Route Table

Your configuration should include one internet gateway and one route table rule that allows your

Kubernetes cluster to access the internet through the internet gateway

The route table rule should have a destination CIDR block of 00000 and target type of

internet gateway The target should be the internet gateway that you intend to use with your

Kubernetes cluster

Load Balancer

The recommended cluster configuration requires two load balancers Create a private load

balancer for your etcd nodes and a public load balancer for your Kubernetes masters Populate

the following table with your load balancerrsquos information to refer to throughout the guide

In the guide the public IP address of your Kubernetes master load balancer is referred to as

loadbalancer_public_ip

Use the following values

Load Balancer

Name

Load

Balancer

Type

Subnet1 Subnet2 Public IP Private IP

Compute

Resource

Back Ends

lb-etcd Private Not applicable Etcd1

Etcd2

Etcd3

lb-k8smaster Public KubeM1

KubeM2

KubeM3

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 10: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

10 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

VCN

You need a single virtual cloud network (VCN) with the following values

VCN Name CIDR Description

k8sOCIoraclevcncom 1000016 VCN used to host network resources for the

Kubernetes cluster

Subnets

You need a VCN and at least one subnet per availability domain (three subnets total) In a

production configuration we recommend creating an etcd master and worker subnet per

availability domain which would result in nine subnets For a test or learning deployment intended

to be removed later creating three subnets (one per availability domain) is sufficient The following

values describe the recommended configuration for a cluster in a region with three availability

domains

Use the following values

Subnet Name CIDR Availability

Domain Description

publicETCDSubnetAD1sub 10020024 AD1 Subnet used for etcd host in AD1

publicETCDSubnetAD2sub 10021024 AD2 Subnet used for etcd host in AD2

publicETCDSubnetAD3sub 10022024 AD3 Subnet used for etcd host in AD3

publicK8SMasterSubnetAD1sub 10030024 AD1 Subnet used for Kubernetes

masters in AD1

publicK8SMasterSubnetAD2sub 10031024 AD2 Subnet used for Kubernetes

masters in AD2

publicK8SMasterSubnetAD3sub 10032024 AD3 Subnet used for Kubernetes

masters in AD3

publicK8SWorkerSubnetAD1sub 10040024 AD1 Subnet used to host Kubernetes

workers in AD1

publicK8SWorkerSubnetAD2sub 10041024 AD2 Subnet used to host Kubernetes

workers in AD2

publicK8SWorkerSubnetAD3sub 10042024 AD3 Subnet used to host Kubernetes

workers in AD3

11 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Security Lists

A production configuration should include the following security lists

etcd_security_list

k8sMaster_security_list

k8sWorker_security_list

For a list of the recommended security rules for each security list see ldquoAppendix A Security

Rulesrdquo

Internet Gateway and Route Table

Your configuration should include one internet gateway and one route table rule that allows your

Kubernetes cluster to access the internet through the internet gateway

The route table rule should have a destination CIDR block of 00000 and target type of

internet gateway The target should be the internet gateway that you intend to use with your

Kubernetes cluster

Load Balancer

The recommended cluster configuration requires two load balancers Create a private load

balancer for your etcd nodes and a public load balancer for your Kubernetes masters Populate

the following table with your load balancerrsquos information to refer to throughout the guide

In the guide the public IP address of your Kubernetes master load balancer is referred to as

loadbalancer_public_ip

Use the following values

Load Balancer

Name

Load

Balancer

Type

Subnet1 Subnet2 Public IP Private IP

Compute

Resource

Back Ends

lb-etcd Private Not applicable Etcd1

Etcd2

Etcd3

lb-k8smaster Public KubeM1

KubeM2

KubeM3

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 11: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

11 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Security Lists

A production configuration should include the following security lists

etcd_security_list

k8sMaster_security_list

k8sWorker_security_list

For a list of the recommended security rules for each security list see ldquoAppendix A Security

Rulesrdquo

Internet Gateway and Route Table

Your configuration should include one internet gateway and one route table rule that allows your

Kubernetes cluster to access the internet through the internet gateway

The route table rule should have a destination CIDR block of 00000 and target type of

internet gateway The target should be the internet gateway that you intend to use with your

Kubernetes cluster

Load Balancer

The recommended cluster configuration requires two load balancers Create a private load

balancer for your etcd nodes and a public load balancer for your Kubernetes masters Populate

the following table with your load balancerrsquos information to refer to throughout the guide

In the guide the public IP address of your Kubernetes master load balancer is referred to as

loadbalancer_public_ip

Use the following values

Load Balancer

Name

Load

Balancer

Type

Subnet1 Subnet2 Public IP Private IP

Compute

Resource

Back Ends

lb-etcd Private Not applicable Etcd1

Etcd2

Etcd3

lb-k8smaster Public KubeM1

KubeM2

KubeM3

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 12: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

12 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute

Populate the following table to better track your Oracle Cloud Infrastructure resources for use with

this guide You can use any Compute instance shape for your compute resources If you are

exploring this technology via this guide we recommend choosing small VM shapes such as

VMStandard11 and VMStandard12

For more information about creating Compute instances see the Compute section of the Oracle

Cloud Infrastructure documentation

Compute

Instance

Name

Compute

Instance Shape

Availability

Domain Subnet

Private IP

Address

Public IP

Address

Role in

Kubernetes

Cluster

Etcd1 AD1 etcd node

Etcd2 AD2 etcd node

Etcd3 AD3 etcd node

KubeM1 AD1 Kubernetes

master

KubeM2 AD2 Kubernetes

master

KubeM3 AD3 Kubernetes

master

KubeW1 AD1 Kubernetes

worker

KubeW2 AD2 Kubernetes

worker

KubeW3 AD3 Kubernetes

worker

In the guide the values in the preceding table are referred to by the following names

Compute Instance Name Private IP Address Public IP Address

Etcd1 etcd1_private_ip etcd1_public_ip

Etcd2 etcd2_private_ip etcd2_public_ip

Etcd3 etcd3_private_ip etcd3_public_ip

KubeM1 kubem1_private_ip kubem2_public_ip

KubeM2 kubem2_private_ip kubem2_public_ip

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 13: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

13 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Compute Instance Name Private IP Address Public IP Address

KubeM3 kubem3_private_ip kubem3_public_ip

KubeW1 kubew1_private_ip kubew1_public_ip

KubeW2 kubew2_private_ip kubew2_public_ip

KubeW3 kubew3_private_ip kubew3_public_ip

Step 2 Set Up a Certificate Authority and Create TLS Certificates

This step has three parts install CFSSL create certificates and copy the certificate files to the

host

Install CFSSL

To generate certificates for use throughout the Kubernetes cluster use the TLS certificate

generating tool from CloudFlare CFSSL Run the commands in this section on your local

computer (running macOS)

1 Download the cfssl package by using curl

curl -O httpspkgcfsslorgR12cfssl_darwin-amd64

2 Modify the permissions on the directory to make cfssl executable

chmod +x cfssl_darwin-amd64

3 Move cfssl to your usrlocalbin to add it to your path

sudo mv cfssl_darwin-amd64 usrlocalbincfssl

4 Download the cfssljson binary by using curl

curl -O httpspkgcfsslorgR12cfssljson_darwin-amd64

5 Modify the permissions on the directory to make cfssjson executable

chmod +x cfssljson_darwin-amd64

6 Move cfssljson into your path

sudo mv cfssljson_darwin-amd64 usrlocalbincfssljson

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 14: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

14 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Create Certificates

This section guides you through setting up the certificates that you need for your cluster Run the

commands in this section on your local computer (running macOS)

1 Create a configuration file that describes your certificate authority (CA) This file will be

used as the authority for all keys related to the cluster

cat gt ca-configjson ltltEOF

signing

default

expiry 8760h

profiles

kubernetes

usages [signing key encipherment server auth client

auth]

expiry 8760h

EOF

2 Create a CA certificate signing request

cat gt ca-csrjson ltltEOF

CN Kubernetes

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU CA

ST Oregon

]

EOF

3 Generate a CA certificate and private key

cfssl gencert -initca ca-csrjson | cfssljson -bare ca

The following files are created

ca-keypem

capem

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 15: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

15 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 Create a client and server TLS certificate signing request for each Kubernetes worker

node Replace $(instance) with a name for the individual Kubernetes worker you are

working with (for example kubeW1 kubeW2 kubeW3)

for instance in worker-0 worker-1 worker-2 do

cat gt $instance-csrjson ltltEOF

CN systemnode$instance

key

algo rsa

size 2048

names [

C US

L Portland

O systemnodes

OU Kubernetes The Hard Way

ST Oregon

]

EOF

5 Generate a certificate and private key for each Kubernetes worker node Run the

following command once for each worker Replace $(instance) with a name for the

individual Kubernetes worker you are working with (for example kubeW1 kubeW2

kubeW3) Replace $(EXTERNAL_IP) and $INTERNAL_IP) with the public (external) and

private (internal) IP addresses of the worker you are working with

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-hostname=$instance$EXTERNAL_IP$INTERNAL_IP

-profile=kubernetes

$instance-csrjson | cfssljson -bare $instance

done

The following files are created

kubeW1-keypem

kubeW1pem

kubeW2-keypem

kubeW2pem

kubeW3-keypem

kubeW3pem

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 16: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

16 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

6 Create the Admin Role Based Access Control (RBAC) certificate signing request The

Admin client certificate is used when you connect to the API server (master) via the

admin role This allows certain privileges under Kubernetes native RBAC

cat gt admin-csrjson ltltEOF

CN admin

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemmasters

OU Cluster

ST Oregon

]

EOF

7 Generate the Admin client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

admin-csrjson | cfssljson -bare admin

The following files are created

admin-keypem

adminpem

8 Create the kube-proxy client certificate signing request This set of certificates is used by

kube-proxy to connect to the Kubernetes master

cat gt kube-proxy-csrjson ltltEOF

CN systemkube-proxy

hosts []

key

algo rsa

size 2048

names [

C US

L Portland

O systemnode-proxier

OU Cluster

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 17: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

17 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

ST Oregon

]

EOF

9 Generate the kube-proxy client certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kube-proxy-csrjson | cfssljson -bare kube-proxy

The following files are created

kube-proxy-keypem

kube-proxypem

10 Create the Kubernetes server certificate Replace kubeMn_private_ip with your

masters private IP addresses and loadbalancer_public_ip with your load balancer IP

address

cat gt kubernetes-csrjson ltltEOF

CN kubernetes

hosts [

103201

ldquokubeM1_private_ip

ldquokubeM2_private_iprdquo

ldquokubeM3_private_iprdquo

loadbalancer_public_ip

127001

kubernetesdefault

]

key

algo rsa

size 2048

names [

C US

L Portland

O Kubernetes

OU Cluster

ST Oregon

]

EOF

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 18: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

18 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

11 Generate the Kubernetes server certificate and private key

cfssl gencert

-ca=capem

-ca-key=ca-keypem

-config=ca-configjson

-profile=kubernetes

kubernetes-csrjson | cfssljson -bare kubernetes

The following files are created

kubernetes-keypem

kubernetespem

12 Create etcd certificates as follows

A Download the following project and follow the instructions to create certificates for

each etcd node Be sure to enter the private IP addresses for your etcd nodes in the

req-csrpem file under config

httpsgithubcomcoreosetcdtreev321hacktls-setup

B Copy your etcd certificates from the cert directory into their own directory called

etcd-certs You will need this path later so put the directory somewhere easy to

remember (for example ~etcd-certs)

C Use the rename command to rename your certificate authority files for etcd to reflect

that they are for etcd

brew install rename

rename scaetcd-ca

Copy CA Files to Hosts

1 Use a script similar to the following one to copy the necessary certificate files to the

hosts Replace node_public_IP with public IP addresses for workers masters and etcd

nodes

Filename copyCAssh

for host in kubeW1_public_IP kubeW2_public_IP kubeW3_public_IP do

scp -i ~sshid_rsa capem kube-proxypem kube-proxy-keypem

ubuntu$host~

done

for host in kubeM1_public_IPrsquo kubeM2_public_IPrsquo kubeM3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem kubernetes-keypem kubernetespem

ubuntu$host~

done

for host in etcd1_public_IP etcd2_public_IPrsquo etcd3_public_IPrsquo do

scp -i ~sshid_rsa capem ca-keypem ubuntu$host~

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 19: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

19 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

done

2 Copy individual worker certificates to the corresponding workers Each worker needs to

have all the etcd certificates the kubeconfig files the capem file and its own certificates

After the files have been copied to the workers each worker should contain files similar to the

following ones

bootstrapkubeconfig etcd1csr etcd1pem etcd2-keypem etcd3csr etcd3pem etcd-ca-keypem

kube-proxy-keypem kube-proxypem kubew2-csrjson kubew2kubeconfig proxy1csr proxy1pem

capem etcd1-keypem etcd2csr etcd2pem etcd3-keypem etcd-cacsr etcd-capem

kube-proxykubeconfig kubew2csr kubew2-keypem kubew2pem proxy1-keypem

To organize these files create a directory called etcd-certs in which to keep the etcd certificates

You can create this directory and move the appropriate files to it by using the following commands

mkdir etcd-certs

mv etcd-certs

Step 3 Bootstrap an HA etcd Cluster

This step has three parts install Docker create an etcd directory and provision the etcd cluster

Run the commands in this section on the etcd nodes unless otherwise instructed

Install Docker on All Nodes

This guide uses the Docker installation script provided at getdockercom You can use another

method to install Docker if you prefer

1 Use curl to download the script

curl -fsSL getdockercom -o get-dockersh

2 To run the script to install Docker run the following command

sh get-dockersh

Create an etcd directory

Run the following command to create a directory for etcd to use to store its data

sudo mkdir varlibetcd

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 20: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

20 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you have an issue while making your etcd cluster and need to redeploy your etcd servers you must

delete this directory or etcd will try to start with the old configuration Delete it by using the following

command sudo rm -rf varlibetcd

Provision the etcd Cluster

1 Set the following variables on each etcd node The NAME_n and HOST_n variables provide

the information required for the CLUSTER variable For the HOST_n variables replace

etcdn_private_ip with your etcd nodes private IP addresses

NAME_1=etcd1

NAME_2=etcd2

NAME_3=etcd3

HOST_1= etcd1_private_ip

HOST_2= etcd2_private_ip

HOST_3= etcd3_private_ip

CLUSTER=$NAME_1=https$HOST_12380$NAME_2=https$HOST_22380

$NAME_3=https$HOST_32380

DATA_DIR=varlibetcd

ETCD_VERSION=latest

CLUSTER_STATE=new

THIS_IP=$(curl http169254169254opcv1vnics0privateIp)

THIS_NAME=$(hostname)

2 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

sudo iptables -F

3 Start the etcd containers by running the following code as-is on each node

sudo docker run -d -p 23792379 -p 23802380 --volume=$DATA_DIRetcd-

data --volume=homeubuntuetcd-certsetcetcd --net=host --name etcd

quayiocoreosetcd$ETCD_VERSION usrlocalbinetcd --name

$THIS_NAME --cert-file=etcetcd$THIS_NAMEpem --key-

file=etcetcd$THIS_NAME-keypem --peer-cert-

file=etcetcd$THIS_NAMEpem --peer-key-file=etcetcd$THIS_NAME-

keypem --trusted-ca-file=etcetcdetcd-capem --peer-trusted-ca-

file=etcetcdetcd-capem --peer-client-cert-auth --client-cert-auth -

-initial-advertise-peer-urls https$THIS_IP2380 --listen-peer-

urls --listen-client-urls --advertise-client-urls --initial-cluster-

token etcd-cluster-0 --initial-cluster $CLUSTER --initial-cluster-state

new --data-dir=etcd-data

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 21: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

21 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

NOTE If you get the following error some or all of the variables that you set in step 1 of this

procedure are missing Ensure that all of your variables have been set correctly

docker invalid reference format

4 Verify the clusters successful deployment by running the following command on any etcd

node

sudo docker exec etcd etcdctl --ca-file=etcetcdetcd-capem --cert-

file=etcetcd$THIS_NAMEpem --key-file=etcetcd$THIS_NAME-keypem

cluster-health

5 If the validation step does not work clear your etcd files and try again

sudo docker stop etcd

sudo docker rm etcd

sudo rm -rf $DATA_DIR

Step 4 Set Up RBAC

Run the commands in this section on your local computer running macOS

1 Download and install kubectl

curl -O httpsstoragegoogleapiscomkubernetes-

releasereleasev160bindarwinamd64kubectl

chmod +x kubectl

sudo mv kubectl usrlocalbin

2 Create and distribute the TLS bootstrap token as follows

A Generate a token

BOOTSTRAP_TOKEN=$(head -c 16 devurandom | od -An -t x | tr -d )

B Generate a token file

cat gt tokencsv ltltEOF

$BOOTSTRAP_TOKENkubelet-bootstrap10001systemkubelet-bootstrap

EOF

C Distribute the token to each master Replace controllern with the public IP address

of each Kubernetes master

for host in controller0 controller1 controller2 do

scp -i ~sshid_rsa tokencsv ubuntu$host~

done

3 Create the bootstrap kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 22: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

22 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--server=https$LB_IP6443

--kubeconfig=bootstrapkubeconfig

kubectl config set-credentials kubelet-bootstrap

--token=$BOOTSTRAP_TOKEN

--kubeconfig=bootstrapkubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kubelet-bootstrap

--kubeconfig=bootstrapkubeconfig

kubectl config use-context default --kubeconfig=bootstrapkubeconfig

4 Create the kube-proxy kubeconfig file

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

5 Distribute the client kubeconfig files Replace kubeWn_public_ip with the public IP

address of each worker node

for host in lsquokubeW1_public_iprsquo lsquokubeW2_public_iprsquo lsquokubeW3_public_iprsquo do

scp -i ~sshid_rsa bootstrapkubeconfig kube-proxykubeconfig

ubuntu$host~

done

Step 5 Bootstrap an HA Kubernetes Control Plane

Provision the Kubernetes masters Run the following commands on the masters

1 Copy the bootstrap token into place

sudo mkdir -p varlibkubernetes

sudo mv tokencsv varlibkubernetes

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 23: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

23 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 If you did not copy the necessary certificates to the Kubernetes masters in ldquoStep 2 Set

Up a Certificate Authority and Create TLS Certificatesrdquo do that now You need the

capem ca-keypem kubernetes-keypem and kubernetespem certificates

To secure communication between the Kubernetes API server (on the masters) and

kubectl (used to control Kubernetes from another machine) and the kubelet (on the

workers) the TLS certificates created in Step 2 are used Communication between the

Kubernetes API server and etcd is also secured via TLS certificates created in Step 2

3 Copy the TLS certificates to the Kubernetes configuration directory

sudo mv capem ca-keypem kubernetes-keypem kubernetespem

varlibkubernetes

4 Download the official Kubernetes release binaries by using wget

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-apiserver

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-controller-manager

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kube-scheduler

wget httpsstoragegoogleapiscomkubernetes-

releasereleasev170binlinuxamd64kubectl

5 Install the Kubernetes binaries

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl

sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl

usrbin

Kubernetes API Server

1 To better organize the certificates place all etcd certificates in their own directory by

using the following commands

sudo mkdir varlibetcd

sudo cp varlibetcd

2 Capture the internal IP address of the machine

INTERNAL_IP=$(curl http169254169254opcv1vnics0privateIp)

3 Drop iptables

Kubernetes and etcd make modifications to the iptables during setup This step drops

iptables completely to allow etcd to set them up as needed

NOTE This step does pose some security risk Ensure that your security rules for your networking

resources are sufficiently locked down before performing this step

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 24: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

24 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

sudo iptables -F

4 Create the systemd unit file for the Kubernetes API server This file instructs systemd on

Ubuntu to manage the Kubernetes API server as a systemd service

cat gt kube-apiserverservice ltltEOF

[Unit]

Description=Kubernetes API Server

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-apiserver

--admission-

control=NamespaceLifecycleLimitRangerServiceAccountDefaultStorageClass

ResourceQuota

--advertise-address=$INTERNAL_IP

--allow-privileged=true

--apiserver-count=3

--audit-log-maxage=30

--audit-log-maxbackup=3

--audit-log-maxsize=100

--audit-log-path=varlibauditlog

--authorization-mode=RBAC

--bind-address=$INTERNAL_IP

--client-ca-file=varlibkubernetescapem

--enable-swagger-ui=true

--etcd-cafile=varlibetcdetcd-capem

--etcd-certfile=varlibetcdetcd3pem

--etcd-keyfile=varlibetcdetcd3-keypem

--etcd-servers=httpsltetcd1 private ipgt2379httpsltetcd2 private

ipgt2379httpsltetcd3 private ipgt2379

--event-ttl=1h

--experimental-bootstrap-token-auth

--insecure-bind-address=0000

--kubelet-certificate-authority=varlibkubernetescapem

--kubelet-client-certificate=varlibkuberneteskubernetespem

--kubelet-client-key=varlibkuberneteskubernetes-keypem

--kubelet-https=true

--runtime-config=rbacauthorizationk8siov1alpha1

--kubelet-preferred-address-

types=InternalIPExternalIPLegacyHostIPHostname

--service-account-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--service-node-port-range=30000-32767

--tls-cert-file=varlibkuberneteskubernetespem

--tls-private-key-file=varlibkuberneteskubernetes-keypem

--token-auth-file=varlibkubernetestokencsv

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 25: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

25 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

5 Start the kube-apiserver service

sudo mv kube-apiserverservice etcsystemdsystemkube-apiserverservice

sudo systemctl daemon-reload

sudo systemctl enable kube-apiserver

sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver --no-pager

6 If your service reports an error debug by using the following commands to ensure that it

is bound to the ports it needs

journalctl -xe

netstat -na | more

netstat -na | grep 6443

7 Run the following command to verify that your Kubernetes API Server is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Scheduler

1 Create the systemd unit file for the Kubernetes scheduler This file instructs systemd on

Ubuntu to manage the Kubernetes scheduler as a systemd service

kube-schedulerservice

cat gt kube-schedulerservice ltltEOF

[Unit]

Description=Kubernetes Scheduler

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-scheduler

--leader-elect=true

--master=http$INTERNAL_IP8080

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-scheduler service

sudo mv kube-schedulerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-scheduler

sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler --no-pager

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 26: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

26 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

3 Run the following command to verify that the Kubernetes scheduler is running

kubectl get componentstatuses

The output from the command should look as follows

Kubernetes Controller Manager

1 Create the systemd unit file for the Kubernetes controller manager This file instructs

systemd on Ubuntu to manage the Kubernetes controller manager as a systemd service

cat gt kube-controller-managerservice ltltEOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-controller-manager

--address=0000

--allocate-node-cidrs=true

--cluster-cidr=102000016

--cluster-name=kubernetes

--cluster-signing-cert-file=varlibkubernetescapem

--cluster-signing-key-file=varlibkubernetesca-keypem

--leader-elect=true

--master=http$INTERNAL_IP8080

--root-ca-file=varlibkubernetescapem

--service-account-private-key-file=varlibkubernetesca-keypem

--service-cluster-ip-range=10320016

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

EOF

2 Start the kube-controller-manager service

sudo mv kube-controller-managerservice etcsystemdsystem

sudo systemctl daemon-reload

sudo systemctl enable kube-controller-manager

sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager --no-pager

3 Run the following command to verify that the Kubernetes controller manager is running

kubectl get componentstatuses

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 27: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

27 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output from the command should look as follows

Step 6 Add a Worker

This step has the following parts generate a kubeconfig file for each worker generate a

kubeconfig file for kube-proxy configure the files on the workers and install several components

Generate a kubeconfig File for Each Worker

Be sure to use the client certificate (created in ldquoStep 2 Set Up a Certificate Authority and Create

TLS Certificatesrdquo) that matches each workerrsquos node name

Use the following script to generate a kubeconfig file for each worker node

for instance in worker-0 worker-1 worker-2 do

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=$instancekubeconfig

kubectl config set-credentials systemnode$instance

--client-certificate=$instancepem

--client-key=$instance-keypem

--embed-certs=true

--kubeconfig=$instancekubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=systemnode$instance

--kubeconfig=$instancekubeconfig

kubectl config use-context default --kubeconfig=$instancekubeconfig

done

The following files are created

worker-0kubeconfig

worker-1kubeconfig

worker-2kubeconfig

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 28: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

28 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Generate a kubeconfig File for Kube-Proxy

Use the following commands to generate a kubeconfig file for kube-proxy to use to connect to the

master

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

--kubeconfig=kube-proxykubeconfig

kubectl config set-credentials kube-proxy

--client-certificate=kube-proxypem

--client-key=kube-proxy-keypem

--embed-certs=true

--kubeconfig=kube-proxykubeconfig

kubectl config set-context default

--cluster=kubernetes-the-hard-way

--user=kube-proxy

--kubeconfig=kube-proxykubeconfig

kubectl config use-context default --kubeconfig=kube-proxykubeconfig

Configure the Files on the Workers

Log in to each worker and run the following file move commands

sudo mkdir varlibkubernetes

sudo mkdir varlibkubelet

sudo mkdir varlibkube-proxy

sudo mv bootstrapkubeconfig varlibkubelet

sudo mv kube-proxykubeconfig varlibkube-proxy

sudo mv capem varlibkubernetes

sudo mv $(hostname)-keypem $(hostname)pem varlibkubelet

sudo mv $(hostname)kubeconfig varlibkubeletkubeconfig

sudo mv capem varlibkubernetes

Install Flannel

1 Install Flannel by using the following commands

wget httpsgithubcomcoreosflannelreleasesdownloadv062flanneld-

amd64 -O flanneld ampamp chmod 755 flanneld

sudo mv flanneld usrbinflanneld

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 29: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

29 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Configure the flannel service by creating the following

etcsystemdsystemflanneldservice file

[Unit]

Description=Flanneld overlay address etcd agent

[Service]

Type=notify

EnvironmentFile=-usrlocalbinflanneld

ExecStart=usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https100172379https10026237

9 --etcd-prefix=kubenetwork

Restart=on-failure

3 Start the flannel service

sudo systemctl daemon-reload

sudo systemctl restart flanneld

sudo systemctl enable flanneld

sudo systemctl status flanneld --no-pager

The last command checks the status of the service and should give output similar to the

following example

ubuntukubew1~$ sudo systemctl status flanneld --no-pager

flanneldservice - Flanneld overlay address etcd agent

Loaded loaded (etcsystemdsystemflanneldservice static vendor

preset enabled)

Active active (running) since Fri 2017-09-15 174006 UTC 1s ago

Main PID 1904 (flanneld)

Tasks 10

Memory 106M

CPU 292ms

CGroup systemsliceflanneldservice

1904 usrbinflanneld -ip-masq=true -iface 100011 --etcd-

cafile=homeubuntuetcd-capem --etcd-certfile=homeubuntuetcd3pem --

etcd-keyfile=homeubuntuetcd3-keypem --etcd-

endpoints=https100072379https1001

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006283241 01904 ipmasqgo47] Adding iptables rule -

s 102000016 -d 102000016 -j MASQUERADE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285301 01904 managergo246] Lease acquired 1020063024

Sep 15 174006 kubew1 systemd[1] Started Flanneld overlay address etcd

agent

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285666 01904 networkgo58] Watching for L3 misses

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006285698 01904 networkgo66] Watching for new subnet leases

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297509 01904 networkgo153] Handling initial subnet events

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297538 01904 devicego163] calling GetL2List()

devlinkIndex 3

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 30: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

30 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297617 01904 devicego168] calling

NeighAdd 10009 866467a629e5

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297682 01904 devicego168] calling NeighAdd 10029

e6762b98c6ab

Sep 15 174006 kubew1 flanneld[1904]

I0915 174006297728 01904 devicego168] calling NeighAdd 10008

a66722ab2f8b

Install the Container Networking Interface (CNI)

1 Create the following files for the CNI service

etcsystemdsystemcni-bridgeservice

[Unit]

Requires=networktarget

Before=dockerservice

[Service]

Type=oneshot

ExecStart=usrlocalbincni-bridgesh

RemainAfterExit=true

usrlocalbincni-bridgesh

binbash

set -x

sbinip link add name cni0 type bridge

sbinip addr add $(grep ^FLANNEL_SUBNET runflannelsubnetenv | cut -

d= -f2) dev cni0

sbinip link set dev cni0 up

2 Run the following block of commands to install and start the CNI service

sudo su

mkdir -p optcnibin etccninetd

chmod +x usrlocalbincni-bridgesh

curl -L --

retry 3 httpsgithubcomcontainernetworkingcnireleasesdownloadv05

2cni-amd64-v052tgz -o tmpcni-plugintargz

tar zxf tmpcni-plugintargz -C optcnibin

printf n name podnetn type flanneln delegate

n isDefaultGateway truen nn gtetccninetd10-

flannelconf

chmod +x usrlocalbincni-bridgesh

systemctl enable cni-bridge ampamp systemctl start cni-bridge

exit

3 Run the following command to check the status of the service

sudo systemctl status cni-bridge

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 31: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

31 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

The output should be similar to the following example

ubuntukubew3~$ sudo systemctl status cni-bridgeservice

cni-bridgeservice

Loaded loaded (etcsystemdsystemcni-bridgeservice static vendor

preset enabled)

Active active (exited) since Fri 2017-09-15 182025 UTC 27s ago

Process 1940 ExecStart=usrlocalbincni-bridgesh (code=exited

status=0SUCCESS)

Main PID 1940 (code=exited status=0SUCCESS)

Sep 15 182025 kubew3 systemd[1] Starting cni-bridgeservice

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link add name cni0

type bridge

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ grep ^FLANNEL_SUBNET

runflannelsubnetenv

Sep 15 182025 kubew3 cni-bridgesh[1940] ++ cut -d= -f2

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip addr add

1020063124 dev cni0

Sep 15 182025 kubew3 cni-bridgesh[1940] + sbinip link set dev cni0

up

Sep 15 182025 kubew3 systemd[1] Started cni-bridgeservice

ubuntukubew1~$ ifconfig

cni0 Link encapEthernet HWaddr 322a0abe35a2

inet addr10200631 Bcast0000 Mask2552552550

inet6 addr fe80302aafffebe35a264 ScopeLink

UP BROADCAST RUNNING MULTICAST MTU1500 Metric1

RX packets0 errors0 dropped0 overruns0 frame0

TX packets8 errors0 dropped0 overruns0 carrier0

collisions0 txqueuelen1000

RX bytes0 (00 B) TX bytes648 (6480 B)

Install Docker

1 Install Docker as follows

wget httpsgetdockercombuildsLinuxx86_64docker-1126tgz

tar -xvf docker-1126tgz

sudo cp dockerdocker usrbin

2 Create the Docker service by creating the following

etcsystemdsystemdockerservice file

[Unit]

Description=Docker Application Container Engine

Documentation=httpdocsdockerio

After=networktarget firewalldservice cni-bridgeservice

Requires=cket cni-bridgeservice

[Service]

ExecStart=usrbindockerd

--bridge=cni0

--iptables=false

--ip-masq=false

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 32: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

32 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--host=unixvarrundockersock

--insecure-registry=registryoracledxcom

--log-level=error

--storage-driver=overlay

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

3 Start the Docker service

sudo systemctl daemon-reload

sudo systemctl enable docker

sudo systemctl start docker

sudo docker version

sudo docker network inspect bridge

4 Ensure that Docker is properly configured by running the following command

sudo docker network inspect bridge

The output should show that Docker is configured to use the cni0 bridge

Install Kubelet

1 Use the following commands to download kubelet version 174

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubectl

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kube-proxy

wget -q --show-progress --https-only --

timestamping httpsstoragegoogleapiscomkubernetes-

releasereleasev174binlinuxamd64kubelet

2 Install kubelet

chmod +x kubectl kube-proxy kubelet

sudo mv kubectl kube-proxy kubelet usrbin

3 Create the following etcsystemdsystemkubeletservice file

etcsystemdsystemkubeletservice

[Unit]

Description=Kubernetes Kubelet

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

After=dockerservice

Requires=dockerservice

[Service]

ExecStart=usrbinkubelet

--allow-privileged=true

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 33: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

33 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

--cluster-dns=1032010

--cluster-domain=clusterlocal

--container-runtime=docker

--image-pull-progress-deadline=2m

--kubeconfig=varlibkubeletkubeconfig

--require-kubeconfig=true

--network-plugin=cni

--pod-cidr=1020088024

--serialize-image-pulls=false

--register-node=true

--runtime-request-timeout=10m

--tls-cert-file=varlibkubeletkubew2pem

--tls-private-key-file=varlibkubeletkubew2-keypem

--hostname-override= kubew2sub08071631352kubevcnoraclevcncom

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

4 Drop iptables before starting the kubelet service

iptables -F

5 Start the kubelet service

sudo systemctl daemon-reload

sudo systemctl enable kubelet

sudo systemctl start kubelet

sudo systemctl status kubelet --no-pager

Install kube-proxy

1 Create the followingetcsystemdsystemkube-proxyservice file for the kube-proxy

service

[Unit]

Description=Kubernetes Kube Proxy

Documentation=httpsgithubcomGoogleCloudPlatformkubernetes

[Service]

ExecStart=usrbinkube-proxy

--cluster-cidr=102000016

--kubeconfig=varlibkube-proxykube-proxykubeconfig

--proxy-mode=iptables

--v=2

Restart=on-failure

RestartSec=5

[Install]

WantedBy=multi-usertarget

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 34: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

34 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

2 Start the kube-proxy service

sudo systemctl daemon-reload

sudo systemctl enable kube-proxy

sudo systemctl start kube-proxy

sudo systemctl status kube-proxy --no-pager

Remove a worker

If you need to remove a worker you can do so by using the following commands Replace

nodename with the name of your Kubernetes worker node from kubectl get nodes

1 Remove any pods from the node

kubectl drain nodename

2 Remove the worker node

kubectl delete node nodename

NOTE These actions do not delete the instance If you want to delete the Oracle Cloud Infrastructure

Compute instance you must do that via the CLI or Console

Step 7 Configure kubectl for Remote Access

kubectl is the command line tool used to control and manage Kubernetes clusters By installing

and configuring kubectl on your local computer you can manage your Kubernetes clusters easily

through your computer rather than logging in to the cluster or some other remote location to

manage the clusters If you want to manage your Kubernetes cluster from a computer other than

your local one run these steps on that computer

This step enables you to connect to your cluster in Oracle Cloud Infrastructure Run the following

commands on your local computer replacing $LB_IP with your load balancerrsquos IP address

kubectl config set-cluster kubernetes-the-hard-way

--certificate-authority=capem

--embed-certs=true

--server=https$LB_IP6443

kubectl config set-credentials admin

--client-certificate=adminpem

--client-key=admin-keypem

kubectl config set-context kubernetes-the-hard-way

--cluster=kubernetes-the-hard-way

--user=admin

kubectl config use-context kubernetes-the-hard-way

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 35: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

35 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Step 8 Deploy Kube-DNS

1 Deploy the kube-dns cluster add-on

kubectl create -f httpsstoragegoogleapiscomkubernetes-the-hard-

waykube-dnsyaml

The output should be similar to the following example

Service account kube-dns created configmap kube-dns created service

kube-dns created deployment kube-dns created

2 List the pods created by the kube-dns deployment

kubectl get pods -l k8s-app=kube-dns -n kube-system

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE

kube-dns-3097350089-gq015 33 Running 0 20s

kube-dns-3097350089-q64qc 33 Running 0 20s

Step 9 Smoke Test

This section walks you through a quick smoke test to ensure the cluster is working as expected

1 Create an nginx deployment with three replicas by using the following command

kubectl run nginx --image=nginx --port=80 --replicas=3

The output should look as follows

deployment nginx created

2 Run the following command to see the pods that your deployment created and ensure

that they are in a Running state

kubectl get pods -o wide

The output should be similar to the following example

NAME READY STATUS RESTARTS AGE IP NODE

nginx-158599303-btl44 11 Running 0 18s 1099495 k8s-

worker-ad1-0

nginx-158599303-ndxtc 11 Running 0 18s 1099493 k8s-

worker-ad2-0

nginx-158599303-r2801 11 Running 0 18s 1099494 k8s-

worker-ad3-0

3 Create a service to connect to your nginx deployment

kubectl expose deployment nginx --type NodePort

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 36: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

36 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

4 View the service that you created Note that --type=LoadBalancer is not currently

supported in Oracle Cloud Infrastructure

kubectl get service

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes 102101 ltnonegt 443TCP 1h

nginx 102162159 ltnodesgt 8030050TCP 1h

At this point you must either manually create a load balancer in BMC that routes to the

cluster (in this example that would be 10216215980) or expose the node port (in this

case 30050) publicly via Oracle Cloud Infrastructurersquos security lists In this guide you do

the latter

5 Modify the workerrsquos security list allowing ingress traffic to 30050

6 Get the NodePort that was set up for the nginx service

NodePort=$(kubectl get svc nginx --output=jsonpath=range

specports[0]nodePort)

7 Get the worker_public_ip value for one of the workers from the UI

8 Test the nginx service with these values by using curl

curl http$worker_public_ip$NodePort

The output should look like the following example

ltDOCTYPE htmlgt

lthtmlgt

ltheadgt

lttitlegtWelcome to nginxlttitlegt

ltstylegt

body

width 35em

margin 0 auto

font-family Tahoma Verdana Arial sans-serif

ltstylegt

ltheadgt

ltbodygt

lth1gtWelcome to nginxlth1gt

ltpgtIf you see this page the nginx web server is successfully installed

and

working Further configuration is requiredltpgt

ltpgtFor online documentation and support please refer to

lta href=httpnginxorggtnginxorgltagtltbrgt

Commercial support is available at

lta href=httpnginxcomgtnginxcomltagtltpgt

ltpgtltemgtThank you for using nginxltemgtltpgt

ltbodygt

lthtmlgt

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 37: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

37 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

Appendix A Security Rules

This appendix outlines the security rules for the following security lists

etcd etcd_security_listsl

Kubernetes masters k8sMaster_security_listsl

Kubernetes workers k8sWorker_security_listsl

The specific ingress and egress rules for each of these security lists are provided in the following

tables All rules are stateful

ETCD SECURITY LIST INGRESS RULES

Source IP Protocol Source Port Range Destination Port Range

1000016 TCP All ALL

1000016 TCP All 22

1000016 TCP All 2379-2380

ETCDSECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

KUBERNETES MASTER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 All All All

1000016 TCP All 3389

1000016 TCP All 6443

1000016 TCP All 22

KUBERNETES MASTER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 38: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

38 KUBERNETES ON ORACLE CLOUD INFRASTRUCTURE

KUBERNETES WORKER SECURITY LIST INGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

1000016 TCP All All

1000016 TCP All 22

1000016 UDP All 30000-32767

KUBERNETES WORKER SECURITY LIST EGRESS RULES

Destination IP Protocol Source Port Range Destination Port Range

00000 All All All

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom

Page 39: Kubernetes on Oracle Cloud Infrastructure · Container orchestrators aim to solve problems with scaling these applications. Container Images ... engine, such as Docker, the container

Oracle Corporation World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone +16505067000

Redwood Shores CA 94065 USA Fax +16505067200

Copyright copy 2018 Oracle andor its affiliates All rights reserved This document is provided for information purposes only and the

contents hereof are subject to change without notice This document is not warranted to be error-free nor subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties and conditions of merchantability or fitness for a particular purpose We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document This document may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without our prior written permission Oracle and Java are registered trademarks of Oracle andor its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group 0218 Kubernetes on Oracle Cloud Infrastructure February 2018 Author Kaslin Fields

C O N N E C T W I T H U S

blogsoraclecomoracle

facebookcomoracle

twittercomoracle

oraclecom