Best Practice - Huawei...2.9 Step 6: Upgrading the Game Application.....44 2.10 Step 7: Deleting...

Cloud Container Engine

Best Practice

Issue 01

Date 2020-09-28

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2020. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 01 (2020-09-28) Copyright © Huawei Technologies Co., Ltd. i

Contents

1 Checklist for Deploying Containerized Applications on the Cloud............................. 1

2 Cluster........................................................................................................................................ 92.1 Creating an IPv4/IPv6 Dual-Stack Cluster on CCE (OBT)..........................................................................................92.2 Implementing High Availability for Containers in CCE............................................................................................182.3 Using a Private Image to Build a Worker Node Image (OBT)..............................................................................222.4 Adding a Salt in the password Field When Creating a Node................................................................................ 262.5 Adding a Second Data Disk to a Node in a CCE Cluster.........................................................................................282.6 Cleaning Up CCE Resources on a Deleted Node........................................................................................................292.7 Changing the Mode of the Docker Device Mapper.................................................................................................. 312.8 Auto Scaling Policies and FAQs for Node Pools......................................................................................................... 362.9 Configuring a Node Scaling Policy..................................................................................................................................37

3 Workload.................................................................................................................................393.1 Properly Allocating Container Computing Resources...............................................................................................393.2 Upgrading Pods Without Interrupting Services.......................................................................................................... 413.3 Modifying Kernel Parameters Using a Privileged Container..................................................................................433.4 Initializing a Container........................................................................................................................................................ 453.5 Setting Time Zone Synchronization................................................................................................................................463.6 Performing Graceful Deletion...........................................................................................................................................51

4 Networking............................................................................................................................. 534.1 Selecting a Network Model When Creating a Cluster on CCE..............................................................................534.2 Planning CIDR Blocks for a CCE Cluster........................................................................................................................584.3 Configuring Containers to Access Public Networks.................................................................................................. 644.4 Implementing Sticky Session Through Load Balancing...........................................................................................674.5 Connecting Two VPCs Through a VPN (VPC Router Networking)...................................................................... 734.6 Allowing Nodes Outside a Cluster in the Same VPC to Access the Pod IP Addresses in the Cluster...... 774.7 Allowing Containers and IDCs to Communicate with Each Other Through VPC, Cloud Connect, andDirect Connect...............................................................................................................................................................................834.8 Obtaining the Source IP Address of a Client for a Container................................................................................864.9 Increasing the Listening Queue Length by Configuring Container Kernel Parameters................................88

5 Storage.....................................................................................................................................915.1 Creating and Attaching Cloud Storage Volumes in a CCE Cluster.......................................................................915.2 Expanding the Disk Capacity of a Node in a CCE Cluster...................................................................................... 96

Cloud Container EngineBest Practice Contents

Issue 01 (2020-09-28) Copyright © Huawei Technologies Co., Ltd. ii

5.3 Mounting an Object Storage Bucket of a Third-Party Tenant.............................................................................. 97

6 Domain Name Resolution................................................................................................. 1036.1 Configuring Domain Name Resolution for CCE Containers................................................................................ 103

7 Charts..................................................................................................................................... 1107.1 Interconnecting CCE with Helm.................................................................................................................................... 1107.2 Installing nginx-ingress Using a Chart........................................................................................................................ 1157.3 Using Kubeflow and Volcano to Train an AI Model............................................................................................... 1177.3.1 Introduction.......................................................................................................................................................................1187.3.2 Implementing Typical Distributed AI Training Tasks...........................................................................................121

8 Permissions........................................................................................................................... 1258.1 Configuring kubeconfig to Implement Refined Management on Cluster Resources................................. 125

9 API&kubectl..........................................................................................................................1309.1 Connecting to Multiple Clusters Using kubectl........................................................................................................1309.2 Using kubectl to Modify hostAliases to Configure Container Parameters..................................................... 132

10 Monitoring..........................................................................................................................13510.1 Connecting a CCE Cluster to Heapster for Monitoring.......................................................................................135

11 Migrating Applications from a Third-Party Cloud Cluster to HUAWEI CLOUDCCE............................................................................................................................................. 13911.1 Preparation......................................................................................................................................................................... 13911.2 Migration Scheme Overview........................................................................................................................................ 14011.3 Creating and Configuring a CCE Hybrid Cluster....................................................................................................14211.4 Migrating Data..................................................................................................................................................................14411.4.1 Migrating Databases and Storage.......................................................................................................................... 14411.4.2 Migrating Container Images..................................................................................................................................... 14511.5 Migrating the Application............................................................................................................................................. 14511.5.1 Preparing Object Storage and Velero.................................................................................................................... 14511.5.2 Backing Up Kubernetes Objects of the ACK Cluster.........................................................................................14711.5.3 Restoring Kubernetes Objects in the Created CCE Cluster.............................................................................14711.5.4 Update and Adaptation.............................................................................................................................................. 14811.5.5 Debugging and Starting the Application..............................................................................................................14811.6 Others.................................................................................................................................................................................. 149

12 Containerizing an Enterprise Application (ERP)....................................................... 15012.1 Overview............................................................................................................................................................................. 15012.2 Containerizing an Entire Application.........................................................................................................................15312.3 Containerization Process............................................................................................................................................... 15512.4 Analyzing the Application............................................................................................................................................. 15612.5 Preparing the Application Runtime........................................................................................................................... 15812.6 Compiling a Startup Script............................................................................................................................................16012.7 Compiling the Dockerfile...............................................................................................................................................16112.8 Building and Uploading an Image............................................................................................................................. 162


Issue 01 (2020-09-28) Copyright © Huawei Technologies Co., Ltd. iii

12.9 Creating a Container Workload.................................................................................................................................. 164

13 Containerizing a Game Application (WOW)............................................................. 16913.1 Overview............................................................................................................................................................................. 16913.2 Deployment Process........................................................................................................................................................ 17213.3 Preparation: Reconstructing the Game Application Architecture....................................................................17313.4 Analyzing the Game Application................................................................................................................................ 17513.5 Preparing the Environment...........................................................................................................................................17613.6 Deploying the Game Application............................................................................................................................... 18013.7 Running the Game.......................................................................................................................................................... 18813.8 Scaling the Game Application..................................................................................................................................... 18913.9 Upgrading the Game Application...............................................................................................................................19213.10 Deleting Resources........................................................................................................................................................ 19313.11 FAQs................................................................................................................................................................................... 194

14 Installing, Deploying, and Interconnecting Jenkins with SWR and CCE Clusters.................................................................................................................................................... 19614.1 Overview............................................................................................................................................................................. 19614.2 Installing and Deploying Jenkins................................................................................................................................ 19714.3 Interconnecting Jenkins with SWR............................................................................................................................. 20314.4 Interconnecting Jenkins with CCE Clusters..............................................................................................................20514.5 Appendix............................................................................................................................................................................. 20914.5.1 Complete Pipeline Script for Image Build, Pushing, and Deployment....................................................... 20914.5.2 Interconnecting Jenkins with RBAC of Kubernetes Clusters.......................................................................... 21014.5.3 Publishing an HTTPS Ingress to ELB...................................................................................................................... 214


Issue 01 (2020-09-28) Copyright © Huawei Technologies Co., Ltd. iv

1 Checklist for Deploying ContainerizedApplications on the Cloud

OverviewSecurity, efficiency, stability, and availability are common requirements on allcloud services. To meet these requirements, the system availability, data reliability,and O&M stability must be perfectly coordinated. This checklist describes thecheck items for deploying containerized applications on the cloud to help youefficiently migrate services to CCE, reducing potential cluster or applicationexceptions caused by improper use.

Check Items

Table 1-1 System availability

Category

Check Item Type

Impact FAQ & Example

Cluster

Before creating acluster, properlyplan the nodenetwork andcontainer networkbased on servicerequirements toallow subsequentservice expansion.

Networkplanning

If the subnet orcontainer CIDRblock where thecluster resides issmall, the numberof available nodessupported by thecluster may be lessthan required.

● Network Planning● Planning CIDR

Blocks for a CCECluster

● How Do I Set theVPC CIDR Blockand Subnet CIDRBlock for a CCECluster?

Cloud Container EngineBest Practice

1 Checklist for Deploying Containerized Applicationson the Cloud

Issue 01 (2020-09-28) Copyright © Huawei Technologies Co., Ltd. 1

https://support.huaweicloud.com/en-us/usermanual-vpc/vpc_0001.html

https://support.huaweicloud.com/en-us/cce_faq/cce_faq_00144.html





Category

Check Item Type


Before creating acluster, properlyplan CIDR blocksfor the relatedDirect Connect,peeringconnection,container network,service network,and subnet toavoid IP addressconflicts.

Networkplanning

If CIDR blocks arenot properly setand IP addressconflicts occur,service access willbe affected.

● Connectivity● Planning CIDR

Blocks for a CCECluster

● Connecting TwoVPCs Through aVPN (VPC RouterNetworking)

● AllowingContainers andIDCs toCommunicate withEach OtherThrough VPC,Cloud Connect, andDirect Connect

When a cluster iscreated, thedefault securitygroup isautomaticallycreated and boundto the cluster. Youcan set customsecurity grouprules based onservicerequirements.

Deployment

Security groups arekey to securityisolation. Impropersecurity policyconfiguration maycause security risksand serviceconnectivityproblems.

● Security GroupOverview

● How Do I Hardenthe VPC SecurityGroup Rules forCCE Cluster Nodes?

Enable the multi-master nodemode, and set thenumber of masternodes to 3 whencreating a cluster.

Reliability

After the multi-master node modeis enabled, threemaster nodes willbe created. If amaster node isfaulty, the clustercan still beavailable withoutaffecting servicefunctions. Incommercialscenarios, it isadvised to enablethe multi-masternode mode.

How Do I CheckWhether a Cluster Isan HA Cluster?Once a cluster iscreated, the number ofmaster nodes cannotbe changed. Exercisecaution when settingthe number of masternodes.




https://support.huaweicloud.com/en-us/vpc_faq/faq_connection.html

https://support.huaweicloud.com/en-us/usermanual-vpc/en-us_topic_0073379079.html









Category

Check Item Type


CCE supportscontainerd andDocker as thecontainer runtime.They areapplicable todifferent scenarios.When creating acluster, select aproper containerruntime based onservicerequirements.

Deployment

Once a cluster iscreated, thecontainer runtimecannot be changedunless the clusteris re-created.

How Do I Select aContainer Runtime?

When creating acluster, select aproper networkmodel, such ascontainer tunnelnetwork or VPCnetwork.

Deployment

After a cluster iscreated, thenetwork modelcannot bechanged. Exercisecaution whenselecting anetwork model.

Selecting a NetworkModel When Creatinga Cluster on CCE

Workload

When creating aworkload, youneed to set theCPU and memorylimits to improveservice robustness.

Deployment

When multipleapplications aredeployed on thesame node, if theupper and lowerresource limits arenot set for anapplication,resource leakageoccurs. As a result,resources cannotbe allocated toother applications,and the applicationmonitoringinformation will beinaccurate.

How Do I Set theUpper and LowerLimits of CPU andMemory Resources fora Workload?











Category

Check Item Type


When creating aworkload, you canset probes forcontainer healthcheck, includingliveness probeand readinessprobe.

Reliability

If the health checkfunction is notconfigured, a podcannot detectservice exceptionsor automaticallyrestart the serviceto restore it. Thisresults in asituation where thepod status isnormal but theservice in the podis abnormal.

● Setting HealthCheck for aContainer

● Enabling ICMPSecurity GroupRules

When creating aworkload, select aproper accessmode (Service).Currently, thefollowing fourtypes of Servicesare supported:ClusterIP,NodePort,LoadBalancer, andDNAT.

Deployment

Improper Serviceconfiguration maycause logicconfusion forinternal andexternal access andresource waste.

● Network Overview

When creating aworkload, do notset the number ofreplicas for asingle pod. Set aproper nodescheduling policybased on yourservicerequirements.

Reliability

For example, if thenumber of replicasof a single pod isset, the service willbe abnormal whenthe node or pod isabnormal. Toensure that yourpods can besuccessfullyscheduled, ensurethat the node hasidle resources forcontainerscheduling afteryou set thescheduling rule.

How Do I Set theNumber of WorkloadPods?




https://support.huaweicloud.com/en-us/usermanual-cce/cce_01_0112.html










Category

Check Item Type


Properly setaffinity and anti-affinity.

Reliability

If affinity and anti-affinity are bothconfigured for anapplication thatprovides Servicesexternally, Servicesmay fail to beaccessed after theapplication isupgraded orrestarted.

Scheduling PolicyOverviewNegative example:For application A,nodes 1 and 2 are setas affinity nodes, andnodes 3 and 4 are setas anti-affinity nodes.Application A exposesa Service through theELB, and the ELBlistens to node 1 andnode 2. Whenapplication A isupgraded, it may bescheduled to a nodeother than nodes 1, 2,3, and 4, and it cannotbe accessed throughthe Service.Cause:Scheduling ofapplication A does notneed to meet bothaffinity and anti-affinity policies. A nodewill be selected forapplication A accordingto either of the policies.In this example, thenode selection is basedon the anti-affinityscheduling policy.






Category

Check Item Type


When creating aworkload, set thepre-stopprocessingcommand(Lifecycle > Pre-Stop) to ensurethat the servicesrunning in thepods can becompleted inadvance in thecase of applicationupgrade or poddeletion.

Reliability

If the pre-stopprocessingcommand is notconfigured, thepod will be directlykilled and serviceswill be interruptedduring applicationupgrade.

● Setting ContainerLifecycleParameters

● When Is Pre-stopProcessing Used?

Table 1-2 Data reliability

Category

Check Item Type


Containerdatapersistency

Select a properdata volume typebased on servicerequirements.

Reliability

When a node isfaulty and cannotbe recovered, datain the local diskcannot berecovered.Therefore, you areadvised to usecloud storagevolumes to ensuredata reliability.

● StorageManagement

Backup

Back upapplication data.

Reliability

Data cannot berestored afterbeing lost.

What Storage ClassesDoes CCE Support?What Are theDifferences BetweenThese StorageClasses?

















Table 1-3 O&M reliability

Category

Check Item Type


Project

The quotas of ECS,VPC, subnet, EIP,and EVS resourcesmust meetcustomerrequirements.

Deployment

If the quota isinsufficient,resources will failto be created.Specifically, userswho haveconfigured autoscaling must havesufficient resourcequotas.

● Which ResourceQuotas Should IPay Attention ToWhen Using CCE?

● Constraints

You are notadvised to modifykernel parameters,systemconfigurations,cluster corecomponentversions, securitygroups, and ELB-related parameterson cluster nodes,or install softwarethat has not beenverified.

Deployment

Exceptions mayoccur on CCEclusters orKubernetescomponents on thenode, making thenode unavailablefor applicationdeployment.

For details, see High-Risk Operations andSolutions.Negative example:1. The container

network isinterrupted after thenode kernel isupgraded.

2. The containernetwork isinterrupted after anopen-sourceKubernetes networkadd-on is installedon a node.

3. The /var/paasor /mnt/paas/kubernetesdirectory is deletedfrom a node, whichcauses exceptionson the node.








https://support.huaweicloud.com/en-us/productdesc-cce/cce_productdesc_0005.html




Category

Check Item Type


Do not modifyinformation aboutresources createdby CCE, such assecurity groupsand EVS disks.Resources createdby CCE are labeledcce.

Deployment

CCE clusterfunctions may beabnormal.

Negative example:1. On the ELB console,

a user changes thename of the listenercreated by CCE.

2. On the VPC console,a user modifies thesecurity groupcreated by CCE.

3. On the EVS console,a user deletes oruninstalls data disksmounted to CCEcluster nodes.

4. On the IAM console,a user deletescce_admin_trust.

All the precedingactions will cause CCEcluster functions to beabnormal.

ProactiveO&M

CCE providesmulti-dimensionalmonitoring andalarm reportingfunctions, andsupports basicresourcemonitoring basedon fine-grainedmetrics byinterconnectingwith ApplicationOperationsManagement(AOM). Alarmsallow users tolocate and rectifyfaults as soon aspossible.

Monitoring

If the alarms arenot configured, thestandard ofcontainer clusterperformancecannot beestablished. Whenan exceptionoccurs, you cannotreceive alarms andwill need tomanually locatethe fault.

● Setting AlarmRules

● Viewing Metrics● Cloud Eye Service

(CES)● Configuring APM

Settings forPerformanceBottleneck Analysis













2 Cluster

2.1 Creating an IPv4/IPv6 Dual-Stack Cluster on CCE(OBT)

This function is in the open beta test (OBT) phase.

This section describes how to set up a VPC with IPv6 CIDR block and create acluster and nodes with an IPv6 address in the VPC, so that the nodes can accessthe Internet.

OverviewIPv6 addresses are used to deal with the problem of IPv4 address exhaustion. If aworker node (such as an ECS) in the current cluster uses IPv4, the node can run indual-stack mode after IPv6 is enabled. Specifically, the node has both IPv4 andIPv6 addresses, which can be used to access the intranet or public network.

Application Scenarios● If your application needs to provide Services for users who use IPv6 clients,

you can use IPv6 EIPs or the IPv4 and IPv6 dual-stack function.● If your application needs to both provide Services for users who use IPv6

clients and analyze the access request data, you can use only the IPv4 andIPv6 dual-stack function.

● If internal communication is required between your application systems orbetween your application system and another system (such as the databasesystem), you can use only the IPv4 and IPv6 dual-stack function.

For details about the dual stack, see IPv4 and IPv6 Dual-Stack Network (OBT)and IPv6 EIP (OBT).

Constraints and Limitations● IPv4/IPv6 dual stack is supported only for hybrid clusters of v1.15 and later,

but not for BMS clusters.● Worker nodes and master nodes in Kubernetes clusters use IPv4 addresses to

communicate with each other.

Cloud Container EngineBest Practice 2 Cluster



https://support.huaweicloud.com/en-us/usermanual-vpc/eip_0001.html

● If the Service type is set to LoadBalancer (ELB) or LoadBalancer (DNAT),only IPv4 addresses are supported.

● Only one IPv6 address can be bound to each NIC.

Applying for the OBT

This function is in the OBT in regions CN North-Beijing4 and CN East-Shanghai1.To apply for the OBT, you can click Virtual Private Cloud under Network on themanagement console and click Try now on the page. After your application isapproved, you can experience the IPv4/IPv6 dual-stack network.

Figure 2-1 Participating in the OBT

Step 1: Create a VPC

Before creating your VPCs, determine how many VPCs, the number of subnets,and what IP address ranges you will need. For details, see Network Planning.

NO TE

● The basic operations for IPv4 and IPv6 dual-stack networks are the same as those forIPv4 networks. Only some parameters are different.

● During the OBT, you can experience the dual-stack network only when you select ECSsconfigured with specific flavors in the following regions.

● SN3 ECSs in AZ2 of region CN North-Beijng4

● C3 and M3 ECSs in AZ1 of region CN East-Shanghai1, and C3 ECSs in AZ2 ofregion CN East-Shanghai1

Perform the following operations to create a VPC named vpc-ipv6 and its defaultsubnet named subnet-ipv6.

1. Log in to the HUAWEI CLOUD console.

2. Click in the upper left corner of the management console and select aregion and a project.

3. Under Network, click Virtual Private Cloud.

4. Click Create VPC.

5. Set the VPC and subnet parameters.

When configuring a subnet, select Enable for IPv6 CIDR Block toautomatically allocate an IPv6 CIDR block to the subnet. IPv6 cannot bedisabled after the subnet is created. Currently, you are not allowed to specifya custom IPv6 CIDR block.




Table 2-1 VPC configuration parameters

Parameter Description Example Value

Region Specifies the desired region.Regions are geographic areasthat are physically isolatedfrom each other. The networksinside different regions are notconnected to each other, soresources cannot be sharedacross different regions. Forlower network latency andfaster access to your resources,select the region nearest you.

CN North-Beijing4

Name VPC name. vpc-ipv6

IPv4 CIDRBlock

Specifies the Classless Inter-Domain Routing (CIDR) blockof the VPC. The CIDR block ofa subnet can be the same asthe CIDR block for the VPC(for a single subnet in theVPC) or a subset (for multiplesubnets in the VPC).The following CIDR blocks aresupported:10.0.0.0/8–24172.16.0.0/12–24192.168.0.0/16–24

192.168.0.0/16

EnterpriseProject

When creating a VPC, you canadd the VPC to an enabledenterprise project.An enterprise projectfacilitates project-levelmanagement and grouping ofcloud resources and users. Thename of the default project isdefault.For details about creating andmanaging enterprise projects,see Enterprise ManagementUser Guide.

default



https://support.huaweicloud.com/en-us/usermanual-em/en-us_topic_0108744738.html



Tag(AdvancedSettings)

Specifies the VPC tag, whichconsists of a key and valuepair. You can add a maximumof ten tags for each VPC.The tag key and value mustmeet the requirements listedin Table 2-3.

● Tag key: vpc_key1● Key value: vpc-01

Table 2-2 Subnet parameter description


AZ An AZ is a geographic locationwith independent powersupply and network facilitiesin a region. AZs are physicallyisolated, and AZs in the sameVPC are interconnectedthrough an internal network.

AZ2

Name Specifies the subnet name. subnet-ipv6

IPv4 CIDRBlock

Specifies the IPv4 CIDR blockfor the subnet. This value mustbe within the VPC CIDR range.

192.168.0.0/24

IPv6 CIDRBlock

Select Enable for IPv6 CIDRBlock. An IPv6 CIDR block willbe automatically assigned tothe subnet. IPv6 cannot bedisabled after the subnet iscreated. Currently, you are notallowed to specify a customIPv6 CIDR block.

N/A

AssociatedRoute Table

Specifies the default routetable to which the subnet willbe associated. You can changethe route table to a customroute table.

Default

Advanced Settings

Gateway Specifies the gateway addressof the subnet.This IP address is used tocommunicate with othersubnets.

192.168.0.1




DNS ServerAddress

By default, two DNS serveraddresses are configured. Youcan change them if necessary.When multiple IP addressesare available, separate themwith a comma (,).

100.125.x.x

DHCP LeaseTime

Specifies the period duringwhich a client can use an IPaddress automaticallyassigned by the DHCP server.After the lease time expires, anew IP address will beassigned to the client. If aDHCP lease time is changed,the new lease automaticallytakes effect when half of thecurrent lease time has passed.To make the change takeeffect immediately, restart theECS or log in to the ECS tocause the DHCP lease toautomatically renew. Fordetails, see How Can I Makethe Changed DHCP LeaseTime of a Subnet Take EffectImmediately?

365 days or 300 hours

Tag Specifies the subnet tag, whichconsists of a key and valuepair. You can add a maximumof ten tags to each subnet.The tag key and value mustmeet the requirements listedin Table 2-4.

● Tag key: subnet_key1● Key value: subnet-01

Table 2-3 VPC tag key and value requirements

Parameter

Requirement ExampleValue

Tagkey

● Cannot be left blank.● Must be unique in a VPC.● Can contain a maximum of 36 characters.● Can contain letters, digits, underscores (_), and

hyphens (-).

vpc_key1



https://support.huaweicloud.com/en-us/vpc_faq/en-us_topic_0177255344.html




Parameter

Requirement ExampleValue

Tagvalue

● Can contain a maximum of 43 characters.● Can contain letters, digits, underscores (_), periods

(.), and hyphens (-).

vpc-01

Table 2-4 Subnet tag key and value requirements

Parameter

Requirement Example Value

Tagkey

● Cannot be left blank.● Must be unique for each subnet.● Can contain a maximum of 36 characters.● Can contain letters, digits, underscores (_),

and hyphens (-).

subnet_key1

Tagvalue

● Can contain a maximum of 43 characters.● Can contain letters, digits, underscores (_),

periods (.), and hyphens (-).

subnet-01

6. Click Create Now.

Step 2: Buy a Hybrid ClusterConstraints

During OBT, IPv4/IPv6 dual-stack clusters are supported only in the CN North-Beijing4 region.

Buying a Hybrid Cluster

Log in to the CCE console and buy a hybrid cluster by following the instructionsprovided in Buying a Hybrid Cluster. Pay attention to the following during thepurchase:

1. Configure the network as follows:



https://console.huaweicloud.com/cce2.0?locale=en-us


Figure 2-2 Configuring network settings

– VPC: Select the created VPC vpc-ipv6.– Subnet: Select the created subnet subnet-ipv6.– Network Model: Select Tunnel network or VPC network.– Container Network Segment: Select Automatically select or manually

configure a CIDR block. The value cannot conflict with that of the subnet.The mask of the container CIDR block must be appropriate. It determinesthe number of available nodes in a cluster. A too small mask value willcause the cluster to soon fall short of nodes.

– Service Network Segment: Select Default or Custom. The value cannotconflict with the container or subnet CIDR block.

2. When creating a node, configure the region, AZ, and node specifications asfollows:

Figure 2-3 Configuring the region, AZ, and node specifications



NO TE

During OBT, the dual-stack function is supported only for nodes of the sn3 type inAZ2, CN North-Beijing4.

– Current Region: CN North-Beijing4

– AZ: AZ2

– Specifications: node of the sn3 type

3. When creating a workload, if you select LoadBalancer (ELB) orLoadBalancer (DNAT) for the Service type, only IPv4 is supported.

After the purchase is complete, choose Resource Management > Nodes and clickthe node name. On the page displayed, view the automatically allocated IPv6address.

Step 3: Buy a Shared Bandwidth and Adding an IPv6 Address to It

By default, the IPv6 address can only be used for private network communication.If you want to use this IPv6 address to access the Internet or be accessed by IPv6clients on the Internet, you need to buy a shared bandwidth and add the IPv6address to it.

If you already have a shared bandwidth, you can add the IPv6 address to theshared bandwidth without purchasing one.

Buying a Shared Bandwidth

1. Log in to the HUAWEI CLOUD management console.


3. Choose Service List > Network > Virtual Private Cloud.

4. In the navigation pane on the left, choose Elastic IP and Bandwidth >Shared Bandwidths.

5. In the upper right corner, click Buy Shared Bandwidth. On the displayedpage, configure parameters as prompted.

Table 2-5 Description


Billing Mode Specifies the billing mode of a sharedbandwidth. The billing mode can be:● Yearly/Monthly: You pay for the

bandwidth by year or month beforeusing it. No charges will be incurredfor the bandwidth during its validityperiod.

● Pay-per-use: You pay for thebandwidth based on the amount oftime you use the bandwidth.

Yearly/Monthly




Region Specifies the desired region. Regions aregeographic areas that are physicallyisolated from each other. The networksinside different regions are notconnected to each other, so resourcescannot be shared across differentregions. For lower network latency andfaster access to your resources, select theregion nearest you.

CN North-Beijing4

Billed By Specifies the shared bandwidth billingfactor.

SelectBandwidth.

Bandwidth Specifies the shared bandwidth size inMbit/s. The minimum bandwidth thatcan be purchased is 5 Mbit/s.

10

BandwidthName

Specifies the name of the sharedbandwidth.

Bandwidth-001

EnterpriseProject

When assigning the shared bandwidth,you can add the shared bandwidth to anenabled enterprise project.An enterprise project facilitates project-level management and grouping ofcloud resources and users. The name ofthe default project is default.For details about creating and managingenterprise projects, see EnterpriseManagement User Guide.

default

RequiredDuration

Specifies the required duration of theshared bandwidth to be purchased. Youneed to specify this parameter only inyearly/monthly billing mode.

2 months

6. Click Next to confirm the configurations and then click Buy Now.

Adding an IPv6 Address to a Shared Bandwidth

1. On the Shared Bandwidths page, click More > Add Public IP Address in theOperation column.

Figure 2-4 Adding an IPv6 address to a shared bandwidth

2. Add the IPv6 address to the shared bandwidth.





Figure 2-5 Adding a dual-stack NIC IPv6 address

3. Click OK.

Verifying the Result

Log in to an ECS and ping an IPv6 address on the Internet to verify theconnectivity. ping6 ipv6.baidu.com is used as an example here. The executionresult is displayed in Figure 2-6.

Figure 2-6 Result verification

2.2 Implementing High Availability for Containers inCCE

Basic Principles

To achieve high availability for your CCE containers, you can do as follows:

1. Deploy containers in a cluster.

2. When creating a workload, set the number of pods to be greater than 2.

3. Allow the workload pods to be randomly scheduled to different nodes in thecluster.

4. If multiple AZs and multiple nodes are involved, set custom schedulingpolicies based on site requirements to maximize resource utilization.



ProcedureAssume that there are four nodes in the cluster: demo-92634, demo-01208,demo-30509, and demo-27003. Demo-92634 and demo-01208 are in the same AZ(AZ 1), demo-30509 is in AZ 2, and demo-27003 is in AZ 3.

Step 1 Log in to the CCE console. In the navigation pane, choose Workloads >Deployments. On the page displayed, click Create Deployment.

Step 2 Configure the following parameters and retain the default values for otherparameters.● Workload Name: name of the workload. Set this parameter to nginx-demo.● Instances: Set it to 1 in this example.

Figure 2-7 Creating a workload

Step 3 Click Next. In the dialog box displayed, click Add Container. On the Open SourceImages tab page, select the nginx image and then click OK.

Step 4 Retain the default values for other parameters, and click Next and then Create.

For details about how to create a workload, see Creating a Deployment.

Step 5 In the workload list, click the name of the created workload (nginx-demo in thisexample).

Step 6 On the Pods tab page, click the IP address of the node where the pod resides.Choose Resource Management > Nodes and click the node name. You will seethat the pod is scheduled to a node in AZ 1.

Figure 2-8 AZ information





Step 7 Back to the Deployment details page, on the Scheduling Policies tab page, clickAdd Custom Scheduling Policy. In the Pod Anti-affinity area, click Add Rulenext to Preferred.

Set the parameter as follows to create a preferred anti-affinity rule for the AZ. Fordetails, see Affinity and anti-affinity.● Weight: A larger weight value indicates a higher priority. In this example, set

it to 50.● Topology Key: a default or custom key for the node label that the system

uses to denote a topology domain. A topology key determines the scopewhere the pod should be scheduled to. In this example, set this parameter tofailure-domain.beta.kubernetes.io/zone.

● Label: pod label. You can use the default label app or a custom label. Set it toapp in this example.

● Operator: Four operators are provided for you to configure label matchingrelationships: In, NotIn, Exists, and DoesNotExist. Operators In and NotInallow one or more label values. Operators Exists and DoesNotExist are usedto determine whether a label exists, and do not require a label value. Set it toIn in this example.

● Value: Set it to nginx-demo in this example.

Figure 2-9 Adding a rule

Step 8 Repeat Step 7 to create another preferred anti-affinity rule for the workload. Setthe parameters as follows:● Weight: 50● Topology Key: kubernetes.io/hostname● Label: app● Operator: In● Value: nginx-demo



https://kubernetes.io/docs/concepts/configuration/assign-pod-node/?spm=a2c4g.11186623.2.34.7f4d38f6C1WPWj#affinity-and-anti-affinity


Step 9 Click OK.

On the Scheduling Policies tab page, you will see the scheduling policies youadded.

Step 10 On the Scaling tab page, click under Manual Scaling to add a pod. You willsee that the pod is preferentially scheduled to a node in AZ 3.


Step 11 On the Scaling tab page, click under Manual Scaling to add another pod. Youwill see that the pod is preferentially scheduled to a node in AZ 2.




Step 12 On the Scaling tab page, click under Manual Scaling. You will see that thenew pod is preferentially scheduled to the remaining node in AZ 1.


----End

2.3 Using a Private Image to Build a Worker NodeImage (OBT)

Constraints● This function is in the OBT only in specific regions, for example, AP-Singapore.● This function is available only for clusters of v1.15 or later.

Image OS and Kernel Version Requirements

You have added a dedicated label to the image. Both the label key and value arecce. The image OS version must be EulerOS 2.5 or CentOS 7.6.

Table 2-6 Mappings between clusters, OSs, and kernels

OS Cluster Version Kernel

CentOS Linux release7.6

v1.17.9-r0 3.10.0-1062.12.1.el7.x86_64



OS Cluster Version Kernel

v1.15.11-r1 3.10.0-1062.12.1.el7.x86_64

v1.15.6-r1 3.10.0-1062.1.1.el7.x86_64

EulerOS release 2.5 v1.17.9-r0 3.10.0-862.14.1.5.h428.eulerosv2r7.x86_64

v1.15.11-r1 3.10.0-862.14.1.5.h428.eulerosv2r7.x86_64

v1.15.6-r1 3.10.0-862.14.1.5.h328.eulerosv2r7.x86_64

● When creating an image, follow the instructions in this section to preventunexpected problems.

● To log in to VMs created from base images, users are required to have thesudo root or root permissions.

PreparationNotes

● Components lvm2, conntrack, sudo, NetworkManager, and ntpd are requiredfor creating a private image. Ensure that these components have beeninstalled.

● Before creating an image, you need to create two ECSs and bind an EIP toeach ECS. One ECS functions as the executor, and the other functions as thehost for creating an image. It takes about 10 minutes to create an image,which generates traffic and consumes resources. An EIP is bound to remotelytransfer the installation package and send installation dependencycommands. Recommended ECS specifications: 4 vCPUs and 8 GB memory

● During image build, an agent will be injected. You need to check whether thecreated image is available only in the current region.

● After the image is created, the ECSs will not be deleted. You need to deletethem manually.

● The private image installation package contains the script and dependentcomponents required for installing the node. The package version variesdepending on the cluster version.

● Ensure that TCP port 22 is enabled in the new inbound rule of the securitygroup.

Procedure

Step 1 Uploading the init_envs.conf File

The init_envs.conf file stores the configurations of the VM created from the baseimage. Apply for a server on the ECS console or use an existing server, log in theserver, and upload the init_envs.conf file to the /root directory on the server.

The following is an example of the init_envs.conf file. Set the parameters basedon the description in Table 2-7.



DOMAIN_NAME=''USER_NAME=''PROJECT_NAME=''PROJECT_ID=''IMS_ENDPOINT=''KEY_PAIR_NAME=''IMAGE_NAME=''

Table 2-7 Description of the init_envs.conf file

Parameter Description

DOMAIN_NAME Account that creates an image.

USER_NAME User that creates an image.

PROJECT_NAME Region to which the project belongs.View the region and project ID on the My Credentialspage.

PROJECT_ID Project ID.View the region and project ID on the My Credentialspage.

IMS_ENDPOINT ims.region.myhuaweicloud.comFor details about regions, see Regions and Endpoints.Example value: ims.cn-north-4.myhuaweicloud.com

KEY_PAIR_NAME (Optional) Name of the key pair, which is the same asthe name of the key pair file in the /root directory.

IMAGE_NAME Optional. The default value is the BASIC-NODE-IMG-timestamp.

Step 2 Obtain the key file. (Skip this step if you log in to the server using a password.)

A key file is the authentication file required for creating an ECS. You can useexisting keys or create new keys. For example, log in to the server and upload thekey file named Keypair.pem to the /root directory to create an ECS.

1. Log in to the HUAWEI CLOUD management console.2. Choose Service List > Computing > Elastic Cloud Server.3. In the navigation pane, choose Key Pair. On the page displayed, click Create

Key Pair.4. Enter a key pair name and click OK.5. In the dialog box displayed, click OK.

View and save the key pair. To ensure security, a key pair can be downloadedonly once. Keep the key pair secure for login.For details about how to create a key pair, see Creating a Key Pair.

----End



https://console.huaweicloud.com/iam/?locale=en-us#/mine/apiCredential

https://console.huaweicloud.com/iam/?locale=en-us#/mine/apiCredential

https://developer.huaweicloud.com/en-us/endpoint?IMS

https://support.huaweicloud.com/en-us/usermanual-ecs/en-us_topic_0014250631.html

Creating a Node Image

Step 1 (Optional) Obtain a base image ID from IMS.

NO TE

For details about how to use an image file to generate an image ID, see Appendix.

Step 2 Log in to the server, upload the init_envs.conf and optionally Keypair.pem files tothe /root directory, and set parameters in the init_envs.conf file.

Step 3 Run the image creation script.

Table 2-8 Commands to be run

Site Command

HUAWEICLOUD

Click here to obtain the installation package (for the Singaporeregion only).Decompress the installation package. When executing create.sh inthe node-image/conf directory, add the following five parameters.The following is an example:bash create.sh ${NODE_EIP} ${PASSWORD} ${ECS_PASSWORD} ${ECS_INSTANCE_ID} ${LINUX_ROLE}

The parameters are described as follows:NODE_EIP=${1:-""} #EIP address of the server that creates the imagePASSWORD=${2:-""} #Password for logging in to HUAWEI CLOUD.This password is used to obtain the token and create an IMS image.ECS_PASSWORD=${3:-"} #Password for logging in to the node thatcreates the image. If the key pair mode is used, this parameter is leftblank.ECS_INSTANCE_ID=${4:-""} #Instance ID of the ECS used to createthe imageLINUX_ROLE=${5:-"root"} #The default user is root. If a non-rootuser is used, set the permission as follows:/etc/sudoersUsername ALL=(ALL) NOPASSWD: ALL

Step 4 After the image is created, use the image ID for verification.

----End

Appendix

This operation is required only when an image file is used to generate an imageID. Perform the following operations:

Step 1 Obtain a base image file from a trusted HUAWEI CLOUD image repository.

For details about how to obtain the image ID, see Quickly Importing an ImageFile (Windows).

Step 2 Import the obtained image file to an OBS bucket of your account.



https://cce-statics.ap-southeast-3.obs.ap-southeast-3.myhuaweicloud.com/package/node-image/node-image-v1.17.9-r0.tgz

https://support.huaweicloud.com/en-us/usermanual-ims/ims_01_0341.html

https://support.huaweicloud.com/en-us/usermanual-ims/ims_01_0341.html

Figure 2-14 Importing the image file to an OBS bucket

Step 3 On IMS, click Create Image on the Private Images tab page. Select Image Filefor Source, which is the image file in the OBS bucket. Set the system disk to 40GB, configure other parameters as required, and click Create Now.

Figure 2-15 Creating an image

----End

2.4 Adding a Salt in the password Field When Creatinga Node

When a node is created through the API, you need to add a salt to the passwordfield to safeguard the password. The procedure is as follows:



NO TE

The salt must be set based on the password complexity requirements:● A string of 8–26 characters.● Contains at least three of the following character types: uppercase letters, lowercase

letters, digits, and special characters !@$%^-_=+[{}]:,./?● Cannot contain the username or the username spelled backwards.● Cannot contain the username, the username spelled backwards, or more than two

consecutive characters in the username (for Windows ECSs).

Python example:

1. Generate a salt.python -c "import crypt, getpass, pwd;print crypt.mksalt()"

Command output:$6$KZ2u71CD4JjQneAy

2. Add \ before $ in the salt. Generate a ciphertext password based on theupdated salt.python -c "import crypt, getpass, pwd;print crypt.crypt('test@123','\$6\$KZ2u71CD4JjQneAy')"

Command output:$6$KZ2u71CD4JjQneAy$WF5dsoOjTgc9RD46i46cCL3H92LMEo78s0rHdfSLDE8PW7ylE2ICcxUGF7/8RBbnxW0crgA3ZGNFA0LLgFaYD0

3. Encode the value of the password field using Base64.echo -n '$6$KZ2u71CD4JjQneAy$WF5dsoOjTgc9RD46i46cCL3H92LMEo78s0rHdfSLDE8PW7ylE2ICcxUGF7/8RBbnxW0crgA3ZGNFA0LLgFaYD0' | base64 | tr "\n" " " | sed s/[[:space:]]//g

Command output:JDYkS1oydTcxQ0Q0SmpRbmVBeSRXRjVkc29PalRnYzlSRDQ2aTQ2Y0NMM0g5MkxNRW83OHMwckhkZlNMREU4UFc3eWxFMklDY3hVR0Y3LzhSQmJueFcwY3JnQTNaR05GQTBMTGdGYVlEMA==

Java example:

1. Obtain a random number as the salt. private static String getCharAndNumr(int length) { String val = ""; Random random = new Random(); for (int i = 0; i < length; i++) { // Indicates whether to output letters or digits. String charOrNum = random.nextInt(2) % 2 == 0 ? "char" : "num"; // Character string if ("char".equalsIgnoreCase(charOrNum)) { // Indicates whether an upper-case or lower-case letter is obtained. int choice = random.nextInt(2) % 2 == 0 ? 65 : 97; val += (char) (choice + random.nextInt(26)); } else if ("num".equalsIgnoreCase(charOrNum)) {// Digit val += String.valueOf(random.nextInt(10)); } } return val; }

2. Generate a salt. private static String generateSalt() { String salt; try { salt = "$6$" + getCharAndNumr(16); }catch (Exception e){ salt = defaultSalt; }



return salt; }

3. Generate a ciphertext password based on the salt. public static String getSaltPassword(String password) { if(StringUtils.isBlank(password)) { throw new BizException("password is empty"); }

String salt = generateSalt();

Crypt crypt = new Crypt(); return crypt.crypt(password, salt); }

4. Encode the value of the password field using Base64.(Base64.getEncoder().encodeToString(AddSaltPasswordUtil.getSaltPassword(cceNodeCreateVo.getPassword()).getBytes()))

2.5 Adding a Second Data Disk to a Node in a CCECluster

You can use the pre-installation script feature to configure CCE cluster nodes(ECSs). For details, see Buying a Hybrid Cluster - Advanced KubernetesSettings.

Before using this feature, write a script that can format data disks and save it toyour OBS bucket. Then, inject a command line that will automatically execute thedisk formatting script when the node is up. The script specifies the size of eachdocker data disk (for example, the default docker disk is 100 GB and theadditional disk is 110 GB) and the mount path (/data/code) of the additionaldisk. In this example, the script is named formatdisk.sh. Note that the scriptmust be run by the root user.

NO TE

● When creating a node in a cluster of v1.13.10 or later, if a data disk is not managed byLVM, follow the instructions in this section to format the data disk before adding thedisk. Otherwise, the data disk will still be managed by LVM.

● When creating a node in a cluster of v1.13.10 or earlier, if a data disk is not managed byLVM, format the data disk. Otherwise, either this data disk or the first data disk will bemanaged by LVM, which is not as expected.

Example command line:

cd /tmp;curl -k -X GET OBS bucket address /formatdisk.sh -1 -O;fdisk -l;sleep 30;bash -x formatdisk.sh 100 /data/code;fdisk -l

Example script (formatdisk.sh):

dockerdisksize=$1mountdir=$2systemdisksize=40i=0while [ 20 -gt $i ]; do echo $i; if [ $(lsblk -o KNAME,TYPE | grep disk | grep -v nvme | awk '{print $1}' | awk '{ print "/dev/"$1}' |wc -l) -ge 3 ]; then break else sleep 5 fi; i=$[i+1]



https://support.huaweicloud.com/en-us/usermanual-cce/cce_01_0028.html#cce_01_0028__li1824844253210

https://support.huaweicloud.com/en-us/usermanual-cce/cce_01_0028.html#cce_01_0028__li1824844253210

done all_devices=$(lsblk -o KNAME,TYPE | grep disk | grep -v nvme | awk '{print $1}' | awk '{ print "/dev/"$1}')for device in ${all_devices[@]}; do isRawDisk=$(lsblk -n $device 2>/dev/null | grep disk | wc -l) if [[ ${isRawDisk} > 0 ]]; then # is it partitioned ? match=$(lsblk -n $device 2>/dev/null | grep -v disk | wc -l) if [[ ${match} > 0 ]]; then # already partited [[ -n "${DOCKER_BLOCK_DEVICES}" ]] && echo "Raw disk ${device} has been partition, will skip this device" continue fi else isPart=$(lsblk -n $device 2>/dev/null | grep part | wc -l) if [[ ${isPart} -ne 1 ]]; then # not parted [[ -n "${DOCKER_BLOCK_DEVICES}" ]] && echo "Disk ${device} has not been partition, will skip this device" continue fi # is used ? match=$(lsblk -n $device 2>/dev/null | grep -v part | wc -l) if [[ ${match} > 0 ]]; then # already used [[ -n "${DOCKER_BLOCK_DEVICES}" ]] && echo "Disk ${device} has been used, will skip this device" continue fi isMount=$(lsblk -n -o MOUNTPOINT $device 2>/dev/null) if [[ -n ${isMount} ]]; then # already used [[ -n "${DOCKER_BLOCK_DEVICES}" ]] && echo "Disk ${device} has been used, will skip this device" continue fi isLvm=$(sfdisk -lqL 2>>/dev/null | grep $device | grep "8e.*Linux LVM") if [[ ! -n ${isLvm} ]]; then # part system type is not Linux LVM [[ -n "${DOCKER_BLOCK_DEVICES}" ]] && echo "Disk ${device} system type is not Linux LVM, will skip this device" continue fi fi block_devices_size=$(lsblk -n -o SIZE $device 2>/dev/null | awk '{ print $1}') if [[ ${block_devices_size}"x" != "${dockerdisksize}Gx" ]] && [[ ${block_devices_size}"x" != "${systemdisksize}Gx" ]]; thenecho "np1

w" | fdisk $device mkfs -t ext4 ${device}1 mkdir -p $mountdir echo "${device}1 $mountdir ext4 noatime 0 0" | tee -a /etc/fstab >/dev/null mount $mountdir fidone

2.6 Cleaning Up CCE Resources on a Deleted Node

Application Scenario

If a cluster contains yearly/monthly billed nodes or nodes managed by the cluster,you will be prompted to clean up CCE resources on ECSs when deleting the cluster



or these nodes. If you do not want to clean up CCE resources when deleting acluster or nodes, follow the procedure described here when you want to do so.

NO TICE

Uninstalling an ECS will delete the CCE system user paas and docker resourcesfrom the ECS. To preserve data before the cleanup, make a copy of the data orsubmit a service ticket.

Procedure

Step 1 Log in to the CCE console. In the navigation pane, choose Resource Management> Nodes. In the same row as the node whose CCE resources will be cleaned up,choose More > Remove.

Figure 2-16 Removing a managed node

Step 2 In the Remove from Cluster dialog box, enter REMOVE to confirm the removal ofthe node, and click OK.

Figure 2-17 Confirming the removal of a managed node

Step 3 Read the message displayed on the page and clean up CCE resources by followingthe on-screen instructions.



https://console.huaweicloud.com/ticket/?locale=en-us#/ticketindex/createIndex

Figure 2-18 On-screen instructions for cleaning up CCE resources on a node

----End

2.7 Changing the Mode of the Docker Device MapperCurrently, private CCE clusters use Device Mapper as the Docker storage driver.

Device Mapper is developed based on the kernel framework and supports manyadvanced volume management technologies on Linux.

Docker Device Mapper storage driver leverages the thin provisioning and snapshotcapabilities of this framework to manage images and containers.

For CCE clusters of v1.7.3-r6 or earlier, the Docker Device Mapper is set to theloop-lvm mode by default. By default, Docker generates data and metadata filesin the /var/lib/docker/devicemapper/devicemapper directory. The two files areattached to loop devices and used as block devices. After multiple containers areattached to the files, the performance deteriorates dramatically.

The loop-lvm mode enables you to use Docker out of the box, without additionalconfiguration. This mode is not recommended in the production environment. TheDocker Device Mapper also supports the direct-lvm mode. This mode enables youto use raw partitions (no file systems). In the medium-load and high-densityenvironments, this mode provides better performance.

To ensure system stability, you need to set the Docker Device Mapper to thedirect-lvm mode.

CCE allows you to change the mode of the Device Mapper on VM nodes runningon EulerOS, CentOS, and SUSE.



NO TICE

● Changing the Docker Device Mapper mode on a node requires a data disk.Therefore, in the change process, the system automatically creates a 100 GBSATA disk and binds it to the node. This data disk requires extra fees. For detailson the fee calculation method, see EVS Pricing Details.

● When the Docker Device Mapper mode on a node is changed to direct-lvm,the container and image data on the node will be deleted. Therefore, you mustback up the container and image data of the node to a private imagerepository or open source image repository before changing the mode.

Procedure

Step 1 Check whether the Docker Device Mapper mode on a node is direct-lvm.

Method 1:

1. Log in to a node on which you want to view the Docker Device Mapper mode.2. Enter the following command to view the configuration information under

Storage Driver.docker info– If the values of the Data file and Metadata file parameters under

Storage Driver are /dev/loopx, the Docker Device Mapper mode of thecurrent node is loop-lvm. Change the mode by following Step 2.Example:

– If the values of the Data file and Metadata file parameters underStorage Driver are left blank and the value of Pool Name is vgpaas-thinpool, the Docker Device Mapper mode of the current node is direct-lvm. You do not need to change the mode.



https://www.huaweicloud.com/en-us/pricing.html#/evs

Example:

Method 2:

1. Log in to a node on which you want to view the Docker Device Mapper mode.

2. Enter the following command and check whether the command outputcontains the information listed below:

cat /etc/docker/daemon.json"dm.thinpooldev=/dev/mapper/vgpaas-thinpool"

– If the command output contains the preceding information, the DockerDevice Mapper mode of the current node is direct-lvm. You do not needto change the mode.

– If the command output does not contain the preceding information or amessage indicating that a file such as daemon.json is unavailable isdisplayed, the Docker Device Mapper mode of the current node is notdirect-lvm. Change the mode by following Step 2.

Step 2 (Optional) If no elastic IP address is bound to the node for which the DockerDevice Mapper mode needs to be changed, bind an elastic IP address.



Step 3 Log in to the node with an elastic IP address as the root user.

Step 4 Create a configuration file.

touch config.yaml

Step 5 Copy the following content to the configuration file:user: domainName: username: password: projectName: apiGatewayIp: iamHostname: ecsHostname: evsHostname: swrAddr: defaultPassword: defaultPrivateKey: hosts: - host: <node_ip_01> user: root password: privateKey: serverId: - host: <node_ip_02> user: root password: privateKey: serverId:

Table 2-9 Parameter description

Parameter Description Example

domainName Tenant name -

username User name -

password User password, which isenclosed in quotation marks('' '')

-

projectName Name of the project towhich the to-be-configurednode belongs

cn-north-4

apiGatewayIp IP address of an APIgateway

-

iamHostname Endpoint of the IAM serviceQuery the endpoint throughRegions and Endpoints.

iam.cn-north-4.myhuaweicloud.com

ecsHostname Endpoint of the ECS serviceQuery the endpoint throughRegions and Endpoints.

ecs.cn-north-4.myhuaweicloud.com



https://developer.huaweicloud.com/en-us/endpoint



evsHostname Endpoint of the EVS serviceQuery the endpoint throughRegions and Endpoints.

evs.cn-north-4.myhuaweicloud.com

swrAddr Address of a softwarerepository

-

defaultPassword (Optional) Default loginpassword of a node. Thevalue must be enclosed inquotation marks ('' '').

-

defaultPrivateKey (Optional) Absolute path tothe default key file forlogging in to a node. Thevalue must be enclosed inquotation marks ('' '').

-

hosts Host array structure [1].You can set multiple nodesfor which you want tochange the Device Mappermode. The followingparameters must beincluded: user, password/privateKey, and serverId.For details about the hostarray structure, see Table2-10.

-

Table 2-10 Parameter description about the host array structure


host IP address of the nodefor which you want tochange the DeviceMapper mode. This nodemust be in the samesubnet as the currentlogged-in node.

-

user User name. Set thisparameter to root.

-





password Password for the rootuser on the node forwhich you want tochange the DeviceMapper mode. The valuemust be enclosed inquotation marks ('' '').NOTE

Set either password orprivateKey.

-

privateKey Absolute path to the keyfile of the root user onthe node for which youwant to change theDevice Mapper mode.The value must beenclosed in quotationmarks ('' '').NOTE

Set either password orprivateKey.

-

serverId ID of the ECScorresponding to thenode for which you wantto change the DeviceMapper mode

076311b7-4c05-48f6-ba27-f0cfe29d424f

Step 6 Modify the configuration of the nodes in the cluster.

It takes about 3 to 5 minutes to configure a node.

curl -k https://<swr-address>:20202/swr/v2/domains/op_svc_servicestage/namespaces/op_svc_servicestage/repositories/default/packages/cluster-versions/versions/base/file_paths/cceadm -1 -O;chmod u+x cceadm; ./cceadmbatch-config-docker --conf=./config.yaml

Replace <swr-address> with the address of a software repository, which is thesame as the value of swrAddr in Table 2-9.

----End

2.8 Auto Scaling Policies and FAQs for Node Pools

What policy is used by CCE to perform auto scaling of a node pool?Priority is now supported for node pools. CCE will select a node pool for autoscaling based on the following policies:

1. Use algorithms to determine whether the node pool can meet the conditionsto allow scheduling of a pod in pending state, including whether the node



resources are greater than requested by the pod, and whether the nodeSelect,nodeAffinity, and taints meet the conditions. In addition, the node pools thatfail to be scaled (due to insufficient resources) and are still in the 15-minutecool-down interval are filtered out.

2. If multiple node pools meet the scaling requirements, the system checks thepriority of each node pool and selects the node pool with the highest priorityfor scaling. The value ranges from 0 to 100 and the default priority is 0. Thevalue 100 indicates the highest priority, and the value 0 indicates the lowestpriority.

3. If multiple node pools have the same priority or no priority is configured forthem, the system selects the node pool that will consume the least resourcesbased on the configured VM specification.

4. If the VM specification of multiple node pools is the same but the node poolsare deployed in different AZs, the system randomly selects a node pool totrigger scaling.

How do the scale-in cool-down interval configured in the node pool andthat configured in the autoscaler add-on affect each other?

Scale-in cool-down interval configured in a node pool

This interval indicates the period during which nodes added to the current nodepool after a scale-out operation cannot be deleted. This interval takes effect at thenode pool level.

Scale-in cool-down interval configured in the autoscaler add-on

The interval after a scale-out indicates the period during which the entire clustercannot be scaled in after the autoscaler add-on triggers scale-out (due to theunschedulable pods, metrics, and scaling policies). This interval takes effect at thecluster level.

The interval after a node is deleted indicates the period during which the clustercannot be scaled in after the autoscaler add-on triggers scale-in. This intervaltakes effect at the cluster level.

The interval after a failed scale-in indicates the period during which the clustercannot be scaled in after the autoscaler add-on triggers scale-in. This intervaltakes effect at the cluster level.

If the resources of the preferred node pool are insufficient, will the systemautomatically select another node pool?

Yes.

2.9 Configuring a Node Scaling PolicyIf a cluster node is idle for a period of time (10 minutes by default), scale-in istriggered, and the idle node is deleted.

However, a node cannot be deleted from a cluster if the following pods exist:

1. Pods that do not meet specific requirements set in PodDisruptionBudget



2. Pods with local storage3. Pods that cannot be scheduled to other nodes due to constraints such as

affinity and anti-affinity policies.4. Pods that have the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

annotation5. Pods (except those created by kube-system DaemonSet) that exist in the

kube-system namespace on the node6. Pods that are not created by the controller (Deployment/ReplicaSet/job/

StatefulSet)



3 Workload

3.1 Properly Allocating Container Computing ResourcesIf a node has sufficient memory resources, a container on this node can use morememory resources than requested, but no more than limited. If the memoryallocated to a container exceeds the upper limit, the container is stopped first. Ifthe container continuously uses memory resources more than limited, thecontainer is terminated. If a stopped container is allowed to be restarted, kubeletwill restart it, but other types of run errors will occur.

Scenario 1The node's memory has reached the memory limit reserved for the node. As aresult, OOM killer is triggered.

Solution

You can either scale up the node or migrate the pods on the node to other nodes.

Scenario 2The upper limit of resources configured for the pod is too small. When the actualusage exceeds the limit, OOM killer is triggered.

Solution

Set a higher upper limit for the workload.

ExampleA pod will be created and allocated memory that exceeds the limit. As shown inthe following configuration file of the pod, the pod requests 50 MB memory andthe memory limit is set to 100 MB.

Example YAML file (memory-request-limit-2.yaml):

apiVersion: v1kind: Podmetadata: name: memory-demo-2

Cloud Container EngineBest Practice 3 Workload


spec: containers: - name: memory-demo-2-ctr image: vish/stress resources: requests: memory: 50Mi limits: memory: "100Mi" args: - -mem-total - 250Mi - -mem-alloc-size - 10Mi - -mem-alloc-sleep - 1s

The args parameters indicate that the container attempts to request 250 MBmemory, which exceeds the pod's upper limit (100 MB).

Creating a pod:

kubectl create -f https://k8s.io/docs/tasks/configure-pod-container/memory-request-limit-2.yaml --namespace=mem-example

Viewing the details about the pod:

kubectl get pod memory-demo-2 --namespace=mem-example

In this stage, the container may be running or be killed. If the container is notkilled, repeat the previous command until the container is killed.

NAME READY STATUS RESTARTS AGE memory-demo-2 0/1 OOMKilled 1 24s

Viewing detailed information about the container:

kubectl get pod memory-demo-2 --output=yaml --namespace=mem-example

This output indicates that the container is killed because the memory limit isexceeded.

lastState: terminated: containerID: docker://7aae52677a4542917c23b10fb56fcb2434c2e8427bc956065183c1879cc0dbd2 exitCode: 137 finishedAt: 2020-02-20T17:35:12Z reason: OOMKilled startedAt: null

In this example, the container can be automatically restarted. Therefore, kubeletwill start it again. You can run the following command several times to see howthe container is killed and started:

kubectl get pod memory-demo-2 --namespace=mem-example

The preceding command output indicates how the container is killed and startedback and forth:

stevepe@sperry-1:~/steveperry-53.github.io$ kubectl get pod memory-demo-2 --namespace=mem-example NAME READY STATUS RESTARTS AGE memory-demo-2 0/1 OOMKilled 1 37sstevepe@sperry-1:~/steveperry-53.github.io$ kubectl get pod memory-demo-2 --namespace=mem-example NAME READY STATUS RESTARTS AGE memory-demo-2 1/1 Running 2 40s

Viewing the historical information of the pod:

kubectl describe pod memory-demo-2 --namespace=mem-example



The following command output indicates that the pod is repeatedly killed andstarted.

... Normal Created Created container with id 66a3a20aa7980e61be4922780bf9d24d1a1d8b7395c09861225b0eba1b1f8511 ... Warning BackOff Back-off restarting failed container

3.2 Upgrading Pods Without Interrupting Services

ScenarioYou can use rolling upgrade to upgrade pods without interrupting services.

In this mode, pods are upgraded one by one, not all at once.

PrerequisitesThe workload to be upgraded has at least two pods. If there is only one pod, youare advised to perform the upgrade after manually scaling the workload into twopods.

Figure 3-1 Manually scaling the workload

Procedure

Step 1 Log in to the CCE console. In the navigation pane on the left, choose Workloads> Deployments.

Step 2 In the workload list, click the name of the workload to be upgraded. The workloaddetails page is displayed.

Step 3 On the Scaling tab page, check the value of Maximum Number of UnavailablePods. The value should range from 0 to the number of workload pods. If themaximum number of unavailable pods is the number of workload pods, servicesmay be interrupted.




Figure 3-2 Setting maximum number of unavailable pods

Step 4 Upgrade the workload in rolling upgrade mode.

1. In the workload list, click the name of the workload to be upgraded. Theworkload details page is displayed.

2. On the Upgrade tab page, replace the image or select a new image version.3. Click Submit and OK.

Figure 3-3 Replacing the image or the image version

4. On the Pods tab page, you can view that one pod is being created and thenthe other is being stopped. This ensures that there is always a pod runningand the service is not interrupted during the upgrade.

Figure 3-4 Rolling upgrade

5. Click on the right. If both pods are in the running state, the upgrade issuccessful.

----End



3.3 Modifying Kernel Parameters Using a PrivilegedContainer

PrerequisitesTo access a Kubernetes cluster from a client, you need to use the Kubernetescommand line tool kubectl. For details about how to connect to kubectl, seeConnecting to a Kubernetes Cluster Using kubectl or web-terminal.

Procedure

Step 1 Create a DaemonSet in the background, select the Nginx image, enable thePrivileged Container, configure the lifecycle, and add the hostNetwork field(value: true).

1. Create a DaemonSet file.vi daemonSet.yamlAn example YAML file is provided as follows:

NO TICE

The spec.spec.containers.lifecycle field indicates the command that will berun after the container is started.

{ "kind": "DaemonSet", "apiVersion": "extensions/v1beta1", "metadata": { "name": "daemonset-test", "labels": { "name": "daemonset-test" }, "enable": true }, "spec": { "selector": { "matchLabels": { "name": "daemonset-test" } }, "template": { "metadata": { "labels": { "name": "daemonset-test" }, "enable": true }, "spec": { "hostNetwork": true, "containers": [ { "name": "daemonset-test", "image": "nginx:alpine-perl", "command": ["/bin/sh"], "args": ["-c", "while :; do time=$(date);done"],




"imagePullPolicy": "IfNotPresent", "lifecycle": { "postStart": { "exec": { "command": ["sysctl", "-w", "net.ipv4.tcp_tw_reuse=1"] } } }, "securityContext": { "privileged": true } } ], "imagePullSecrets": [{ "name": "default-secret" }] } } }}

2. Create a DaemonSet.kubectl create –f daemonSet.yaml

Step 2 Check whether the DaemonSet is successfully created.

Kubectl get daemonset DaemonSet name

In this example, run the following command:

Kubectl get daemonset daemonset-test

Information similar to the following is displayed:

NAME DESIRED CURRENT READY UP-T0-DATE AVAILABLE NODE SELECTOR AGEdaemonset-test 2 2 2 2 2 <node> 2h

Step 3 Query the container ID of the DaemonSet.

docker ps -a|grep DaemonSet name


docker ps -a|grep daemonset-test


897b99faa9ce 3e094d5696c1 "/bin/sh -c while..." 31 minutes ago Up 30 minutes ault_fa7cc313-4ac1-11e9-a716-fa163e0aalba_0

Step 4 Access the container.

docker exec -it containerid /bin/sh


docker exec -it 897b99faa9ce /bin/sh

Step 5 Check whether the configured command is executed after the container is started.

sysctl -a |grep net.ipv4.tcp_tw_reuse

If the following information is displayed, the system parameters are modifiedsuccessfully:



net.ipv4.tcp_tw_reus=1

----End

3.4 Initializing a Container

ConceptsBefore containers running applications are started, one or some init containers arestarted first. If there are multiple init containers, they will be started in the definedsequence. The application containers are started only after all init containers runto completion and exit. Storage volumes in a pod are shared. Therefore, the datagenerated in the init containers can be used by the application containers.

Init containers can be used in multiple Kubernetes resources, such asDeployments, DaemonSets, and jobs. They perform initialization before applicationcontainers are started.

ScenarioBefore deploying a service, you can use an init container to make preparationsbefore the pod where the service is running is deployed. After the preparations arecomplete, the init container runs to completion and exit, and the container to bedeployed will be started.

● Scenario 1: Wait for other modules to be ready. For example, an applicationcontains two containerized services: web server and database. The web serverservice needs to access the database service. However, when the application isstarted, the database service may have not been started. Therefore, webserver may fail to access database. To solve this problem, you can use an initcontainer in the pod where web server is running to check whether databaseis ready. The init container runs to completion only when database isaccessible. Then, web server is started and initiates a formal access request todatabase.

● Scenario 2: Initialize the configuration. For example, the init container cancheck all existing member nodes in the cluster and prepare the clusterconfiguration information for the application container. After the applicationcontainer is started, it can be added to the cluster using the configurationinformation.

● Other scenarios: For example, register a pod with a central database anddownload application dependencies.

For details, see Init Containers.

Procedure

Step 1 Edit the YAML file of the init container workload.

vi deployment.yaml

An example YAML file is provided as follows:

apiVersion: apps/v1kind: Deployment



https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

metadata: name: mysqlspec: replicas: 1 selector: matchLabels: name: mysql template: metadata: labels: name: mysql spec: initContainers: - name: getresource image: busybox command: ['sleep 20'] containers: - name: mysql image: percona:5.7.22 imagePullPolicy: Always ports: - containerPort: 3306 resources: limits: memory: "500Mi" cpu: "500m" requests: memory: "500Mi" cpu: "250m" env: - name: MYSQL_ROOT_PASSWORD value: "mysql"

Step 2 Create an init container workload.

kubectl create -f deployment.yaml


deployment.apps/mysql created

Step 3 Query the created docker container.

docker ps -a|grep mysql

The init container will exit after it runs to completion. The query result Exited (0)shows the exit status of the init container.

----End

3.5 Setting Time Zone Synchronization

Case Scenarios● Scenario 1: Setting Time Zone Synchronization Between Containers and

Nodes● Scenario 2: Setting Time Zone Synchronization Among Containers,

Container Logs, and Nodes



● Scenario 3: Setting Time Zone Synchronization Between Workloads andNodes

● Scenario 4: Changing the Time Zone of a Node in the Cluster of an EarlierVersion

Scenario 1: Setting Time Zone Synchronization Between Containers andNodes

Step 1 Log in to the CCE console. In the navigation pane on the left, choose Workloads> Deployments or Workloads > StatefulSets. Click Create Deployment orCreate StatefulSet.

Step 2 On the Specify Basic Info page, enable Time Zone Synchronization so that thesame time zone will be used for both the container and the node.

Figure 3-5 Enabling the time zone synchronization

Step 3 Log in to the node, go to the container, and check whether the time zone of thecontainer is the same as that of the node.

date -R


Tue, 04 Jun 2019 15::08:47 +0800

docker ps -a|grep test


docker exec -it oedd74c66bdb /bin/sh

date -R





Tue, 04 Jun 2019 15:09:20 +0800

----End

Scenario 2: Setting Time Zone Synchronization Among Containers, ContainerLogs, and Nodes

The difference between the time when the Java application prints logs and thecontainer's standard time obtained in date -R mode is 8 hours.

Step 1 Log in to the CCE console. In the navigation pane on the left, choose Workloads> Deployments or Workloads > StatefulSets. Click Create Deployment orCreate StatefulSet.

Step 2 On the Specify Basic Info page, enable Time Zone Synchronization so that thesame time zone will be used for both the container and the node.

Figure 3-6 Enabling the time zone synchronization

Step 3 Log in to the node, go to the container, and modify the catalina.sh script.

cd /usr/local/tomcat/bin

vi catalina.sh

If you cannot run the vi command in the container, go to Step 4 or run the vicommand to add -Duser.timezone=GMT+08 to the script, as shown in thefollowing figure.

Step 4 Copy the script from the container to the node, add -Duser.timezone=GMT+08 tothe script, and then copy the script from the node to the container.

Run the following commands to copy files in the container to the host machine:

docker cp mycontainer: /usr/local/tomcat/bin/catalina.sh /home/catalina.sh

Run the following commands to copy files from the host machine to the container:




docker cp /home/catalina.sh mycontainer:/ usr/local/tomcat/bin/catalina.sh

Step 5 Restart the container.

docker restart container_id

Step 6 Check whether the time zone of the logs is the same as that of the node.

On the CCE console, click the workload name. On the workload details pagedisplayed, click Logs in the upper right corner to view the log details. It takesabout 5 minutes to load the logs.

Figure 3-7 Viewing workload logs

----End

Scenario 3: Setting Time Zone Synchronization Between Workloads andNodes

● Method 1: Set the time zone to CST when creating a container image.● Method 2: If you do not want to modify the container, when creating a

workload on the CCE console, mount the /etc/localtime directory of thelocal host to the /etc/localtime directory of the container.The detailed procedure is as follows:

a. When creating a workload, click Data Storage and then click Add LocalVolume.

b. Select HostPath for Type, set Host Path to /etc/localtime, ContainerPath to /etc/localtime, and Permission to Read-only, leave subPathunspecified, and click OK.




Figure 3-8 Adding a local volume

Scenario 4: Changing the Time Zone of a Node in the Cluster of an EarlierVersion

Symptom

The time zone of the node managed by the user is the UTC time zone by default.

Solution

In clusters of an earlier version (v1.7.3-r10 or earlier), the UTC time zone is usedby default when nodes are managed. In clusters of later versions (v1.9.2-r1), thetime zone has been changed to the local time zone.

Perform the following steps to change the time zone of the node to the local timezone:

Step 1 Upgrade the cluster. After the cluster is upgraded to the latest version, all nodes inthe cluster automatically switch to the local time zone.

1. Log in to the CCE console. In the navigation pane, choose ResourceManagement > Clusters.

2. Click More for the cluster you want to upgrade, and select Upgrade from thedrop-down menu.Then, upgrade the cluster as prompted.

If you do not want to upgrade the cluster, perform the following steps to manuallychange the time zone of the node:

Step 2 (Optional) Log in to each node and manually change the time zone.

1. Log in to the node whose time zone is to be changed as user root.2. Obtains the time zone information.

TIME_ZONE=`cat /var/paas/conf/cluster.conf | python -m json.tool | grep'\"timezone\":' | awk -F '"' '{print $4}'`; echo $TIME_ZONE;

3. Change the time zone of the current node to the time zone specified by$TIME_ZONE.ln -sf /usr/share/zoneinfo/${TIME_ZONE} /etc/localtime

4. Check whether the time zone has taken effect:ls -al /etc/localtime;date;




5. Repeat the preceding steps to change the time zone of other nodes in thecluster.

----End

3.6 Performing Graceful Deletion

What Is Graceful Deletion?When a user requests to delete a resource object that contains a pod (such as anRC or Deployment), if the resource object has requests being processed, theresource object is deleted after the requests are processed.

How to Use Graceful DeletionYou can run Kubernetes commands for graceful deletion. The kubectl deletecommand deletes resources by resource name or label. Example:

kubectl delete po rc-nginx-btv4jkubectl delete po -lapp=nginx-2

When Kubernetes instructs the node to run the docker stop command, Dockersends the system signal SIGTERM to the process whose PID is 1 in the containerand waits for the applications in the container to stop. If the waiting time reachesthe specified timeout (30s by default), Docker will send the system signal SIGKILLto forcibly kill the process.

● To download open source images, run the corresponding command. Forexample, to obtain the Nginx image, run the following command:docker pull nginx

● To download the images that you have uploaded to SWR, perform thefollowing operations:

a. Log in to the SWR console.b. Choose My Images in the navigation pane and then click the image to be

downloaded. The details page is displayed.c. Click the Pull/Push tab, and run the docker pull command to download

the desired image as prompted.

The specified timeout can be overwritten by --grace-period flag. To forcibly deleteresources, you need to specify --force flag and set the period to 0. If a pod is to beforcibly deleted, the scheduler will place a new pod on the node before the nodereleases the pod to be deleted, and the pod to be deleted will be evictedimmediately.

The following example indicates that the pod is forcibly deleted immediately:

kubectl delete po rc-nginx-btv4j --grace-period=0 --force



https://console.huaweicloud.com/swr/?locale=en-us#/app/dashboard

NO TICE

When the delete command is run, the resource version is not checked. If anupdate operation is performed when the delete command is run, the updateoperation is deleted together with the resource.



4 Networking

4.1 Selecting a Network Model When Creating aCluster on CCE

CCE uses Huawei's proprietary high-performance container networking add-ons tosupport the tunnel network, VPC network, and Yangtse network models.

CA UTION

After a cluster is created, the network model cannot be changed. Exercise cautionwhen selecting a network model.

● Tunnel network: The container network is an overlay tunnel network on topof a VPC network and uses the VXLAN technology. This network model isapplicable when there is no high requirements on performance. VXLANencapsulates Ethernet packets as UDP packets for tunnel transmission.Though at some cost of performance, the tunnel encapsulation enableshigher interoperability and compatibility with advanced features (such asnetwork policy-based isolation), meeting the requirements of mostapplications.

Cloud Container EngineBest Practice 4 Networking


Figure 4-1 Container tunnel network

● VPC network: The container network uses VPC routing to integrate with theunderlying network. This network model is applicable to performance-intensive scenarios. The maximum number of nodes allowed in a clusterdepends on the route quota in a VPC network. Each node is assigned a CIDRblock of a fixed size. VPC networks are free from tunnel encapsulationoverhead and outperform container tunnel networks. In addition, as VPCrouting includes routes to node IP addresses and container network segment,container pods in the cluster can be directly accessed from outside the cluster.



Figure 4-2 VPC network

● Yangtse: The container network deeply integrates the native elastic networkinterface (ENI) capability of VPC, uses the VPC CIDR block to allocatecontainer addresses, and supports direct traffic distribution to containersthrough an ELB to deliver high performance.The Yangtse network model is now in the open beta test (OBT) phase. After acluster is created, the network model cannot be changed. Therefore, exercisecaution when selecting the network model.



https://support.huaweicloud.com/en-us/productdesc-vpc/en-us_topic_0013748729.html

Figure 4-3 Yangtse network

The following table lists the differences between the network models.

Table 4-1 Network comparison

Dimension

Tunnel Network VPC Network Yangtse

Corecomponents

OVS IPVlan ENI

Applicableclusters

Hybrid clusterVM cluster

Hybrid clusterVM cluster

CCE Turbo cluster

Supportfornetworkpolicies(networkpolicy)

Yes No You can create asecurity group in aVPC. For details, seeSecurity GroupOverview.





Dimension


Supportfor ENI

No Yes. The containernetwork is deeplyintegrated with theVPC network, andENI is used for podsto communicate.

Yes. ENI is used forpods tocommunicate.

IP addressmanagement

IP addresses canbe migrated.

● Each node isallocated with asmall subnet.

● A static route isadded on the VPCrouter with thenext hop set tothe node IPaddress.

A VPC subnet can becustomized and usedto allocate IPaddresses to pods.

Networkperformance

Performance lossdue to VXLANtunnelencapsulation

● No performanceloss as no tunnelencapsulation isrequired;performancecomparable tobare metalnetworks

● Data forwardedacross nodesthrough the VPCrouter

● The bottom-layerVPC networkcapability is fullyused; nodependency onthe VPC router.

● High-performancedirect connectionbetween the ELBand containers

Networking scale

A maximum of2,000 nodes aresupported.

Limited by the VPCroute table. Fordetails, see UsageRestrictions.

● Cluster scale:unlimited

● The number ofpods on a singlenode is limited bythe number ofENIs supported bythe ECS instance.For details, seeECSSpecifications.The maximumnumber of ENIsmust be greaterthan the numberof add-onsinstalled.



https://support.huaweicloud.com/en-us/productdesc-vpc/overview_0003.html


https://support.huaweicloud.com/en-us/productdesc-ecs/en-us_topic_0159822360.html

https://support.huaweicloud.com/en-us/productdesc-ecs/en-us_topic_0159822360.html

Dimension


Externaldependency

None Static route table ofthe VPC router

ENI capability

Applicationscenarios

● Commoncontainerservicescenarios

● Scenarios thatdo not havehighrequirements onnetwork latencyand bandwidth

● Scenarios thathave highrequirements onnetwork latencyand bandwidth

● Containers cancommunicate withVMs using amicroserviceregistrationframework, suchas Dubbo and CSE.

● Scenarios thathave highrequirements onnetwork latency,bandwidth, andperformance

● Containers cancommunicate withVMs using amicroserviceregistrationframework, suchas Dubbo andCSE.

NO TICE

1. The actual cluster scale is limited by the quota of custom routes of the VPC.Therefore, estimate the number of required nodes before creating a VPC.

2. By default, the VPC network model supports direct communication betweencontainers and hosts in the same VPC. If a peering connection policy isconfigured between the VPC and another VPC, the containers can directlycommunicate with hosts on the peer VPC. In addition, in hybrid networkingscenarios such as cloud private line and VPN, communication betweencontainers and hosts on the peer end can also be achieved with properplanning.

4.2 Planning CIDR Blocks for a CCE ClusterBefore creating a cluster on CCE, determine the number of VPCs, number ofsubnets, container CIDR blocks, and Services for access based on servicerequirements.

This topic describes the functions of various addresses in the CCE cluster in theHUAWEI CLOUD VPC environment and how to plan CIDR blocks. For moreinformation, see Network Planning.

Basic ConceptsVPC CIDR Block

Virtual Private Cloud (VPC) enables you to provision logically isolated,configurable, and manageable virtual networks for cloud servers, cloud containers,




and cloud databases. You have complete control over your virtual network,including selecting your own CIDR block, creating subnets, and configuringsecurity groups. You can also assign EIPs and allocate bandwidth in your VPC forsecure and easy access to your business system. For details, see What Is VirtualPrivate Cloud.

Subnet CIDR Block

A subnet is a network that manages ECS network planes. It supports IP addressmanagement and DNS. The IP addresses of all ECSs in a subnet belong to thesubnet.

Figure 4-4 VPC CIDR block architecture

By default, ECSs in all subnets of the same VPC can communicate with oneanother, while ECSs in different VPCs cannot communicate with each other.

You can create a VPC peering connection to enable ECSs in different VPCs tocommunicate with each other. For details, see VPC Peering Connection.

Container (Pod) CIDR Block

Pod is a Kubernetes concept. Each pod has an IP address.

When creating a cluster on CCE, you can specify the pod (container) CIDR block,which cannot overlap with the subnet CIDR block. For example, if the subnet CIDRblock is 192.168.0.0/16, the container CIDR block cannot be 192.168.0.0/18 or192.168.1.0/18, because these addresses are included in 192.168.0.0/16.

Service CIDR Block

Service is also a Kubernetes concept. Each Service has an address. When creating acluster on CCE, you can specify the Service CIDR block. Similarly, the Service CIDRblock cannot overlap with the subnet CIDR block or the container CIDR block. TheService CIDR block can be used only within a cluster.

For details about the relationship between these CIDR blocks, see Figure 4-5.





https://support.huaweicloud.com/en-us/productdesc-vpc/vpc_Concepts_0011.html

How Do I Select a CIDR Block?Single-VPC Single-Cluster Scenarios

These are the simplest scenarios. The VPC CIDR block is determined when the VPCis created. When creating a CCE cluster, select a CIDR block different from that ofthe current VPC.

Figure 4-5 CIDR block in the single-VPC single-cluster scenario

Single-VPC Multi-Cluster Scenarios

Multiple CCE clusters are created in a VPC.

In the VPC network mode, pod packets are forwarded through VPC routes. CCEautomatically configures a routing table on the VPC routes to each container CIDRblock.

Pay attention to the following:

● The VPC address is determined during VPC creation. When creating a cluster,select a CIDR block for each cluster that does not overlap with the VPC CIDRblock or other container CIDR blocks.

● The container CIDR blocks of all clusters cannot overlap, but the Service CIDRblocks can. In this case, CCE clusters are partially interconnected. A pod of acluster can directly access the pods of another cluster, but cannot access theServices of the cluster.

● The network scale is limited by the VPC route table. For details, see UsageRestrictions.





Figure 4-6 VPC network - multi-cluster scenario

In the tunnel network model, the container network is an overlay network planedeployed over the VPC network. Though at some cost of performance, the tunnelencapsulation enables higher interoperability and compatibility with advancedfeatures (such as network policy-based isolation), meeting the requirements ofmost applications.

Figure 4-7 Tunnel network - multi-cluster scenario



● The container CIDR blocks of all clusters can overlap, so do the Service CIDRblocks.

● It is recommended that ELB be used for the cross-cluster access betweencontainers.

VPC Interconnection Scenarios

When two VPC networks are interconnected, you can configure the packets to besent to the peer VPC in the route table.

In the VPC network model, after creating a peering connection, you need to addroutes for the peering connection to enable communication between the twoVPCs.



Figure 4-8 VPC Network - VPC interconnection scenario

To interconnect cluster containers across VPCs, you need to create VPC peeringconnections. For details, see VPC Peering Connection.



● The container CIDR blocks of all clusters cannot overlap, but the Service CIDRblocks can.

● Add the peer container CIDR block to the route table of the VPC peeringconnection. The following is an example:

Figure 4-9 Adding the peer container CIDR block to the local route on theVPC console

For details, see Creating a VPC Peering Connection with Another VPC inYour Account.



https://support.huaweicloud.com/en-us/productdesc-vpc/vpc_Concepts_0011.html



In the tunnel network model, after creating a peering connection, you need to addroutes for the peering connection to enable communication between the twoVPCs.

Figure 4-10 Tunnel network - VPC interconnection scenario



● The container CIDR blocks of all clusters cannot overlap, but the Service CIDRblocks can.

● Add the peer subnet CIDR block to the route table of the VPC peeringconnection. The following is an example:

Figure 4-11 Adding the subnet CIDR block of the peer cluster node to thelocal route on the VPC console

For details, see Creating a VPC Peering Connection with Another VPC inYour Account.





VPC-IDC Scenarios

Similar to the VPC interconnection scenario, some CIDR blocks in the VPC arerouted to the IDC. The pod IP addresses of CCE clusters cannot overlap with theaddresses within these CIDR blocks. To access the pod IP addresses in the cluster inthe IDC, you need to configure the route table to the private line VBR on the IDC.

4.3 Configuring Containers to Access Public NetworksYou can use the NAT Gateway service to enable container pods in a VPC to accessthe Internet. The NAT Gateway service provides source network addresstranslation (SNAT), which translates private IP addresses to a public IP address bybinding an elastic IP address (EIP) to the gateway, providing secure and efficientaccess to the Internet. Figure 4-12 shows the SNAT architecture. The SNATfunction allows the container pods in a VPC to access the Internet without beingbound to an EIP. SNAT supports a large number of concurrent connections, whichmakes it suitable for applications involving a large number of requests andconnections.

Figure 4-12 SNAT

To enable a container pod to access the Internet, perform the following steps:



https://www.huaweicloud.com/en-us/product/nat.html

Step 1 Buy an EIP.

1. Log in to the management console.


3. Click at the upper left corner and choose Network > Elastic IP in theexpanded list.

4. On the EIPs page, click Buy EIP.

5. Set parameters as required.

NO TE

Set Region to the region where container pods are located.

Figure 4-13 Buying an elastic IP address

Step 2 Buy a NAT gateway. For details, see Buying a NAT Gateway.



3. Click at the upper left corner and choose Network > NAT Gateway inthe expanded list.

4. On the displayed page, click Buy NAT Gateway.

5. Set parameters as required.



https://support.huaweicloud.com/en-us/qs-natgateway/nat_qs_0003.html

NO TE

Select the VPC and subnet that are the same as those of the namespace wherecontainer pods are located.

Figure 4-14 Buying a NAT gateway

Step 3 Configure an SNAT rule and bind the EIP to the subnet. For details, see Adding anSNAT Rule.



3. Click at the upper left corner and choose Network > NAT Gateway inthe expanded list.

4. On the page displayed, click the name of the NAT gateway for which youwant to add the SNAT rule.

5. On the SNAT Rules tab page, click Add SNAT Rule.6. Set parameters as required.

NO TE

Select the subnet that is the same as that of the namespace where container pods arelocated.







Figure 4-15 Adding an SNAT rule

After the SNAT rule is configured, workloads can access public networks from thecontainer. Public networks can be pinged from the container.

----End

4.4 Implementing Sticky Session Through LoadBalancing

Concepts

Session persistence is one of the most common while complex problems in loadbalancing.

Session persistence is also called sticky sessions. After the sticky session function isenabled, requests from the same client are distributed to the same backend ECSby the load balancer for better continuity.

In load balancing and sticky session, connection and session are two key concepts.When only load balancing is concerned, session and connection refer to the samething.

Simply put, if a user needs to log in, it can be regarded as a session; otherwise, aconnection.

The sticky session mechanism fundamentally conflicts with the basic functions ofload balancing. A load balancer forwards requests from clients to multiplebackend servers to avoid overload on a single server. However, sticky sessionrequires that some requests be forwarded to the same server for processing.Therefore, you need to select a proper sticky session mechanism based on theapplication environment.



Prerequisites● Three nodes are available in a cluster and have been bound with elastic IP

addresses.

● Cluster nodes have been connected to kubectl. For details, see Connecting toa Kubernetes Cluster Using kubectl or web-terminal.

Layer-4 Load Balancing (Service)In layer-4 load balancing, source IP address-based sticky session (Hash routingbased on the client IP address) can be enabled. To enable source IP address-basedsticky session on Services, the following conditions must be met:

1. Sticky session is enabled on the ELB backend server (you can add annotationsto the Service to enable sticky session). For details, see How to Use ELB in aCluster.

2. Service Affinity of the Service is set to Node level (that is, the value of theexternalTrafficPolicy field of the Service is Local).

3. Anti-affinity is enabled for the backend applications of the Service. For detailsabout how to enable anti-affinity, see Pod Anti-Affinity.

Layer-7 Load Balancing (Ingress)In layer-7 load balancing, sticky session based on HTTP cookies and app cookiescan be enabled. To enable such sticky session, the following conditions must bemet:

1. Anti-affinity is enabled for the applications (workloads) corresponding to theingress. For details about how to enable anti-affinity, see Pod Anti-Affinity.

2. Node affinity is enabled for the Service corresponding to the ingress. Fordetails about how to enable node affinity, see Node Affinity.





https://support.huaweicloud.com/en-us/api-cce/cce_02_0087.html

https://support.huaweicloud.com/en-us/api-cce/cce_02_0087.html




3. Cookie-based sticky session is enabled on the ELB backend server group(which can be enabled using annotation on the CCE ingress). If app cookiesare used, the backend pods must support cookies.

Procedure

Step 1 Create a Nginx workload.

1. Log in to the CCE console. In the navigation pane on the left, chooseWorkloads > Deployments. Click Create Deployment.

2. Enter a workload name and set Instances to 3.

3. Click Next: Add Container. In the dialog box displayed, click Add Container.On the Open Source Images tab page, select the nginx image and then clickOK.

4. Retain the default values of image parameters and click Next: SetApplication Access to set the workload access type.Click Add Service. In this example, set Access Type to NodePort andContainer Port to 80.




5. Click Next: Configure Advanced Settings, choose Inter-Pod Affinity andAnti-affinity > Anti-affinity with Pods, click Add, and select the currentworkload. Click OK.

Step 2 In the navigation pane on the left, choose Resource Management > Network. Onthe Ingresses tab page, click Create Ingress.

Step 3 Set the ingress parameters.



Figure 4-16 Creating an ingress

Configure the forwarding policy. In ELB Settings, enable the sticky sessionfunction, and click Create.

Figure 4-17 ELB Settings

If the application cookie is selected, the cookie name must be specified.

After the ingress is created, it is displayed in the ingress list.



Figure 4-18 Viewing the created ingress

Step 4 Choose Service List > Network > Elastic Load Balance. Click the name of theload balancer to access the load balancer details page. On the Backend ServerGroups tab page, check whether Sticky Session is Enabled. Ensure that it isenabled.

Figure 4-19 Enabling the sticky session feature

Step 5 Log in to a node (named test in this example) bound with an EIP and run thefollowing commands:

1. Save the cookie.curl -H "Host:www.example.com" http://EIP:80 -c test



2. Access the ingress.curl -H "Host:www.zq.com" http://EIP:80 -b test

3. View logs.kubectl logs podnameThe command output displays only the logs generated after the pod on anode is accessed.

NO TE

The kubernetes.io/elb.session-affinity-option: '{"persistence_timeout":"10"}' key-valuepair is added to the annotations of the ingress. Therefore, 10 minutes after themodification, if you run the curl -H "Host:www.zq.com" http://EIP:80 -b test command toaccess the ingress and then run the kubectl logs podname command, the commandoutput will contain the logs generated after at least one pod is accessed.

----End

4.5 Connecting Two VPCs Through a VPN (VPC RouterNetworking)

PrerequisitesThe cluster's network model is VPC router. The local and peer subnets cannotoverlap. Local subnet CIDR blocks cannot overlap.



Deployment Architecture

Table 4-2 Local VPC

VPN Gateway IP Subnet CIDR Block Container CIDR Block

10.4.126.61 192.168.0.0/16 172.16.0.0/16

Table 4-3 Peer VPC

VPN Gateway IP Subnet CIDR Block

10.112.222.206 172.16.0.0/24

Procedure

Step 1 Create VPN gateways and VPN connections on the local and peer VPCs.


2. Click Service List. Under Network, click Virtual Private Network.

3. On the displayed page, click Create VPN Gateway.

4. On the page displayed, set Billing Mode to Pay-per-use and retain thedefault values for other parameters.

5. In the VPN Connection area, set the local VPN by referring to Figure 4-20.

Figure 4-20 Local VPN

6. Repeat the preceding operations to create a peer VPN.



Figure 4-21 Peer VPN

Step 2 Choose Service List > Computing > Elastic Cloud Server and access the detailspage of the container node. On the NICs tab page, disable Source/DestinationCheck for the local and peer ECSs.

Figure 4-22 Disabling Source/Destination Check

Step 3 Enable the ICMP protocol on the local and peer ECSs.

1. Click the Security Groups tab in Figure 4-22 and click the name of a securitygroup in the list to expand the security group rules.

2. Click Modify Security Group Rule on the right. On the displayed page, clickthe Inbound Rules tab and click Add Rule.




3. Set Protocol Port to ICMP and click OK.

Figure 4-24 Enabling ICMP

Step 4 Add a user-defined route in the local VPC and set the next hop of the containernetwork segment to the IP address of the corresponding node.

1. Choose Service List > Network > Virtual Private Cloud.2. Click the VPC name. On the VPC details page, click Route Tables on the right.

3. Click the name of the target route table. On the page displayed, click AddRoute.

Figure 4-25 Adding a route

Step 5 Return to the CCE console and view the pod IP address on the Pods tab page onthe local workload details page.



Step 6 Log in to the remote ECS and access the container IP address in the local cluster.

----End

4.6 Allowing Nodes Outside a Cluster in the Same VPCto Access the Pod IP Addresses in the Cluster

Background

CCE nodes can directly access the pod IP address of each container through theKubernetes network. However, it is not the common case that VMs outside acluster in the same VPC access the pod IP address of a container in that cluster.

Pod IP addresses are commonly used communication inside a cluster. If externalcommunication is required, Services are usually used.

However, in customer scenarios, for example, when Consul is used, the customer'spod registers its IP address with Consul when being started. As a result, all IPaddresses used in the entire architecture are pod IP addresses. Therefore, it isnecessary to discuss the communication method in such scenarios.

Prerequisites

Nodes outside the cluster and the cluster are in the same VPC.

Scenarios● Scenario 1: VPC Network Model● Scenario 2: Container Tunnel Network (Recommended for Non-

Production Environments)

Scenario 1: VPC Network Model

Procedure

Step 1 Ensure that the network model of the cluster is VPC network.



Log in to the CCE console and choose Resource Management > Clusters. On thecluster list, click the name of the cluster to be operated. On the Cluster Detailspage, check that the network model of the cluster is VPC network.

Figure 4-26 Viewing the network model (VPC network)

Step 2 View and record the pod IP address and the IP address of the node where the podis located.

On the CCE console, view the pod IP address on the Pods tab page on theworkload details page.

Figure 4-27 Viewing the pod IP address

Step 3 Add the ICMP protocol.

1. Choose Service List > Network > Virtual Private Cloud.2. In the navigation pane on the left, choose Access Control > Security Groups.

Click the security group name to view its details.3. On the Inbound Rules tab page, click Add Rule and set Protocol & Port to

ICMP.





Figure 4-28 Adding an inbound rule

Step 4 Log in to a node outside the cluster in the same VPC and access the container IPaddress.

----End

Scenario 2: Container Tunnel Network (Recommended for Non-ProductionEnvironments)

Generally, VMs in the same VPC are connected. Therefore, you can add routes.Specifically, configure a route whose destination address is the container CIDRblock on the VM that needs to access the pod IP address and send the packetto the specified node.

Procedure

Step 1 Ensure that the network model of the cluster is Tunnel network.

Log in to the CCE console and choose Resource Management > Clusters. On thecluster list, click the name of the target cluster. On the Cluster Details page,check that the network model of the cluster is Tunnel network and the containerCIDR block is 172.18.0.0/16, as shown in Figure 4-29.




Figure 4-29 Viewing the network model (tunnel network)

Step 2 View and record the pod IP address and the IP address of the node where the podis located.

On the CCE console, view the pod IP address on the Pods tab page on theworkload details page.

Figure 4-30 Viewing the pod IP address

Step 3 Add a route.

Select a node in the cluster as the gateway. For example, use the IP address192.168.0.74 of the node where the pod is located, as shown in Figure 4-30.

Add a route in either of the following ways:

● For the same subnet of the same VPC, run the native route command on theLinux VM.route add -net 172.18.0.0/16 gw 192.168.0.74

After this command is run, the packets whose destination IP addresses are inthe CIDR block 172.18.0.0/16 are sent to the gateway 192.168.0.74. Thismethod can be used to specific nodes. However, the following error messageis displayed when you add different subnets in the same VPC. This method isnot applicable to hosts in different subnets in the same VPC.




● For different subnets in the same VPC, the native route command cannot beused on the Linux VM. You need to add routes to a route table on theHUAWEI CLOUD VPC console.

a. Choose Service List > Network > Virtual Private Cloud.b. Click the VPC name in the VPC list.c. In the Networking Components area on the right, click the number next

to Route Tables. On the displayed Route Tables page, click the routetable name and then click Add Route under Routes.

d. In the Add Route dialog box, set Destination to 172.18.0.0/16, NextHop Type to Server, and Next Hop to **-**-***(192.168.0.74).

Figure 4-31 Adding a route

As shown in the preceding figure, this method applies to all nodes in theVPC. Compared with the route command in method 1, this method doesnot have a fine granularity.

Step 4 Add firewall rules.

HUAWEI CLOUD ECS has its own firewall and security group rules. Therefore, afteradding a route, you need to enable corresponding security group rules to allowtraffic to pass.

The security group rules to be enabled vary in different scenarios. In this example,HTTP port 80 is required. Therefore, you only need to allow traffic on port 80 inthe inbound rule list of CCE nodes.

1. Choose Service List > Network > Virtual Private Cloud.2. In the navigation pane on the left, choose Access Control > Security Groups.

Click the security group name to view its details.3. On the Inbound Rules tab page, click Add Rule, set Protocol & Port to

Custom TCP and 80.

Figure 4-32 Viewing the security group rule



For security, the CIDR block of the peer end can be narrowed down to the VPCCIDR block.

Step 5 Disable source address verification for ECS NICs.

The ECS NIC verifies the source address. The source address of the packet returnedby the CCE nodes is the pod IP address. Therefore, the packet is intercepted. Youneed to disable the verification function on CCE nodes.

Choose Service List > Computing > Elastic Cloud Server and click the name ofthe target cluster node to view the node details. On the NICs tab page, disableSource/Destination Check for the local and peer ECSs.

Figure 4-33 Disabling Source/Destination Check

Step 6 Perform verification test.

If the preceding steps are performed for the Nginx container on the node whose IPaddress is 192.168.0.74, you can directly access the pod IP address from a nodeoutside the cluster.



Figure 4-34 Pod IP address accessed successfully

----End

Conclusion:

● In the scenario where container tunnel network is used, you need to properlyplan the internal CIDR block and ensure that no virtual CIDR block conflictexists in the internal network.

● Generally, the VPC network works better for a node outside a cluster in thesame VPC to access a pod inside the cluster.

4.7 Allowing Containers and IDCs to Communicatewith Each Other Through VPC, Cloud Connect, andDirect Connect

ScenarioBy using VPC, Cloud Connect, and Direct Connect, IP addresses in the containerCIDR block (172.56.0.0/16) and IDC CIDR block (20.227.0.0/16) can communicatewith each other in the VPC network model.



Figure 4-35 Example network topology

Procedure

Step 1 Create a Direct Connect connection.

1. Log in to the management console, click in the upper left corner, and

select the desired region and project. Click at the upper left corner andchoose Network > Direct Connect in the expanded list.

2. In the navigation pane on the left of the console, choose Direct Connect >Connections. On the displayed page, click Create Connection.

3. On the Create Connection page, click Full Service Installation.



Step 2 Create a virtual gateway.

Choose Direct Connect > Virtual Gateways, and click Create Virtual Gatewayon the right. Add the VPC CIDR block and the container CIDR block in the VPCnetwork model.

Figure 4-36 Creating a virtual gateway

Step 3 Create a Cloud Connect connection.

1. In the navigation pane on the left of the console, choose Cloud Connect >Cloud Connections. On the displayed page, click Create Cloud Connection.

2. After the connection is created, click the cloud connection name to go to itsdetails page. On the Network Instances tab page, click Load NetworkInstance to add VPC information.



3. Check the VPC CIDR blocks on the Cloud Connect VPC and ensure that theVPC and container CIDR blocks have been added.

4. Add the VGW CIDR blocks on the Direct Connect gateway.

5. Check the VGW CIDR blocks on the Direct Connect gateway and ensure thatthe remote subnets are correctly added.

Step 4 Test the connectivity.

1. On an IDC host, traceroute the IP address of the container node or containeron the cloud to check whether the route to the cloud gateway of DirectConnect is normal.

a. If the route is normal, Direct Connect has a return route.b. If the route to the cloud gateway of Direct Connect is abnormal, check

whether the route settings at both ends of Direct Connect are correct.2. If the IP address cannot be tracerouted, try the ping or telnet operation.

Before using ping, ensure that the ICMP policy has been enabled for thesecurity group if the target is a HUAWEI CLOUD ECS.

----End

4.8 Obtaining the Source IP Address of a Client for aContainer

Layer-7 Load Balancing (Ingress)

In layer-7 load balancing mode, the client IP address cannot be obtained fromlayer-4 load balancing (the client IP address cannot be viewed using netstat). Itcan be obtained only from x-forward-for in the HTTP header of layer-7 loadbalancing.



Key test point: Obtain the HTTP request header x-forward-for from thecontainer. The obtained IP address is the IP address of the client.

NodePortSet the affinity of a NodePort Service to Node level instead of Cluster level. Thatis, set spec.externalTrafficPolicy of the Service to Local.

Figure 4-37 Selecting a node-level affinity

LoadBalancerWhen a LoadBalancer Service is used, the following prerequisites must be met toobtain the source IP address from the container:

1. Select Node level instead of Cluster level for Service Affinity.2. Select Source IP hash or install the TOA plug-in of the ELB service.

For details about how to install the TOA plug-in, see Configuring the TOAPlug-in. In the CentOS environment, do not perform the kernel upgradehighlighted in Figure 4-38. This is because upgrading the kernel may causenode unavailability or container network disconnection.



https://support.huaweicloud.com/en-us/usermanual-elb/en-us_elb_06_0001.html

https://support.huaweicloud.com/en-us/usermanual-elb/en-us_elb_06_0001.html

Figure 4-38 Installing the kernel module development package

In this case, you can obtain the source IP address of the client in layer-4 loadbalancing (you can run the netstat command to view the source IP address).

Key test point: In this case, you can run the netstat command to view the IPaddress used by the client to connect to pods.

4.9 Increasing the Listening Queue Length byConfiguring Container Kernel Parameters

Scenario

net.core.somaxconn indicates the maximum number of half-open connectionsthat can be backlogged in a listening queue. The default value is 128. If the queueis overloaded, you need to increase the listening queue length.

Procedure

Step 1 Log in to the node.

● If the node is in a cluster of v1.17:

Edit the /opt/cloud/cce/kubernetes file to enable net.core.somaxconn.

● If the node is in a cluster of v1.15:

Edit the /var/paas/kubernetes/kubelet/kubelet file and enablenet.core.somaxconn.



Step 2 Restart the kubelet.● If the node is in a cluster of v1.13:

systemctl restart kubeletCheck the kubelet status.systemctl status kubelet

● If the node is in a V1.11 cluster:su paas -c '/var/paas/monit/bin/monit restart kubelet'Check the kubelet status.su paas -c '/var/paas/monit/bin/monit summary'

Example printout after kubelet of a node in a V1.13 cluster is started:

Step 3 Log in to the CCE console and create a workload. For details, see Creating aDeployment.

When creating a workload, choose Scheduling Policies > Workload-NodeAffinity and Anti-affinity > Affinity with Node in Set Advanced Settings, andselect the node that you have logged in to in Step 1.

Step 4 Click Create YAML and set the following kernel parameters in bold:apiVersion: apps/v1kind: Deploymentmetadata: annotations: description: '' labels: appgroup: '' name: test1 namespace: defaultspec: replicas: 1 selector: matchLabels: app: test1 template: metadata: annotations: metrics.alpha.kubernetes.io/custom-endpoints: '[{"api":"","path":"","port":"","names":""}]' labels: app: test1






spec: containers: - image: 'nginx:1.14-alpine-perl' name: container-0 resources: requests: cpu: 250m memory: 512Mi limits: cpu: 250m memory: 512Mi imagePullSecrets: - name: default-secret securityContext: sysctls: - name: net.core.somaxconn value: '3000' affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - 192.168.x.x

Step 5 Log in to the node in Step 1, access the container, and check whether theparameter configuration takes effect.

For example, run the following command to view the containerid workloadwhose name is test:

docker ps –a |grep test

Run the following command to access the container:

docker exec -it containerid /bin/sh

Run the following command to check whether the configuration takes effect:

sysctl -a |grep somax

Figure 4-39 Viewing the parameter configuration

----End



5 Storage

5.1 Creating and Attaching Cloud Storage Volumes in aCCE Cluster

PrerequisitesPrepare a VM or node that can run the kubectl commands and configure kubectl.For details, click the Kubectl tab on the CCE cluster details page.

EVSDuring the use of EVS, expected EVS disks can be automatically created andattached. Currently, common I/O, high I/O, and ultra-high I/O are supported,which correspond to sata, sas, and ssd respectively.

Step 1 Log in to an available VM or node.

Step 2 Configure the pvc-evs-auto-example.yaml file for creating a PVC.

touch pvc-evs-auto-example.yaml

vi pvc-evs-auto-example.yaml

Example:

apiVersion: v1kind: PersistentVolumeClaimmetadata: name: pvc-evs-auto-example namespace: default annotations: volume.beta.kubernetes.io/storage-class: sata volume.beta.kubernetes.io/storage-provisioner: flexvolume-huawei.com/fuxivol labels: failure-domain.beta.kubernetes.io/region: cn-north-4 failure-domain.beta.kubernetes.io/zone: cn-north-4aspec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi

Cloud Container EngineBest Practice 5 Storage


Where,

● volume.beta.kubernetes.io/storage-class is the EVS disk type. Currently,high I/O (SAS), ultra-high I/O (SSD), and common I/O (SATA) are supported.

● For volume.beta.kubernetes.io/storage-provisioner, flexvolume-huawei.com/fuxivol must be used.

● failure-domain.beta.kubernetes.io/region indicates the region where thecluster is located.

● failure-domain.beta.kubernetes.io/zone indicates the AZ where the EVS diskis created. It must be the same as the AZ planned for the workload. Fordetails, see Regions and Endpoints.

● storage indicates the storage capacity (unit: Gi).

Step 3 Create a PVC.

./kubectl create -f pvc-evs-auto-example.yaml

Step 4 Configure the evs-pod-example.yaml file to create a pod.

touch evs-pod-example.yaml

vi evs-pod-example.yaml

Example:

apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: evs-pod-example namespace: defaultspec: replicas: 1 selector: matchLabels: app: evs-pod-example template: metadata: labels: app: evs-pod-example spec: containers: - image: nginx name: container-0 volumeMounts: - mountPath: /tmp name: pvc-evs-auto-example restartPolicy: Always volumes: - name: pvc-evs-auto-example persistentVolumeClaim: claimName: pvc-evs-auto-example

Where,

● name indicates the name of the pod to be created.● app indicates the name of a pod workload.● image indicates the download address of an image.● mountPath indicates a mounting path.

Step 5 Create a pod.

./kubectl create -f evs-pod-example.yaml




Step 6 After the pod is created, log in to the CCE console, choose Resource Management> Storage in the navigation pane, and then click the EVS tab to view the bindingrelationship between the workload and PVC.

----End

SFSDuring the use of SFS, expected file systems can be automatically created andattached.


Step 2 Configure the pvc-sfs-auto-example.yaml file to create a PVC.

touch pvc-sfs-auto-example.yaml

vi pvc-sfs-auto-example.yaml

Example:

apiVersion: v1kind: PersistentVolumeClaimmetadata: annotations: volume.beta.kubernetes.io/storage-class: nfs-rw volume.beta.kubernetes.io/storage-provisioner: flexvolume-huawei.com/fuxinfs name: pvc-sfs-auto-example namespace: defaultspec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi

Where,

● name indicates the name of the PVC to be created.● storage indicates the storage capacity (unit: Gi).


./kubectl create -f pvc-sfs-auto-example.yaml

Step 4 Configure the sfs-pod-example.yaml file to create a pod.

touch sfs-pod-example.yaml

vi sfs-pod-example.yaml

Example:

apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: sfs-pod-example namespace: defaultspec: replicas: 1 selector: matchLabels: app: sfs-pod-example template: metadata:



labels: app: sfs-pod-example spec: containers: - image: nginx name: container-0 volumeMounts: - mountPath: /tmp name: pvc-sfs-auto-example restartPolicy: Always volumes: - name: pvc-sfs-auto-example persistentVolumeClaim: claimName: pvc-sfs-auto-example

Where,

● name indicates the name of the pod to be created.● app indicates the name of a pod workload.● image indicates the download address of an image.● mountPath indicates a mounting path.


./kubectl create -f sfs-pod-example.yaml

Step 6 After the pod is created, log in to the CCE console, choose Resource Management> Storage in the navigation pane, and then click the SFS tab to view the bindingrelationship between the workload and PVC.

----End

OBSDuring the use of OBS, expected OBS buckets can be automatically created andattached. Currently, standard and infrequent access OBS buckets are supported,which correspond to obs-standard and obs-standard-ia respectively.


Step 2 Configure the pvc-obs-auto-example.yaml file for creating a PVC.

touch pvc-obs-auto-example.yaml

vi pvc-obs-auto-example.yaml

Example:

apiVersion: v1kind: PersistentVolumeClaimmetadata: annotations: volume.beta.kubernetes.io/storage-class: obs-standard volume.beta.kubernetes.io/storage-provisioner: flexvolume-huawei.com/fuxiobs name: pvc-obs-auto-example namespace: defaultspec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi

Where,



● volume.beta.kubernetes.io/storage-class indicates the bucket specification.Currently, both obs-standard and obs-standard-ia are supported

● name indicates the name of the PVC to be created.

● storage indicates the storage capacity (unit: Gi).


./kubectl create -f pvc-obs-auto-example.yaml

Step 4 Configure the obs-pod-example.yaml file for creating a pod.

touch obs-pod-example.yaml

vi obs-pod-example.yaml

Example:

apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: obs-pod-example namespace: defaultspec: replicas: 1 selector: matchLabels: app: obs-pod-example template: metadata: labels: app: obs-pod-example spec: containers: - image: nginx name: container-0 volumeMounts: - mountPath: /tmp name: pvc-obs-auto-example restartPolicy: Always volumes: - name: pvc-obs-auto-example persistentVolumeClaim: claimName: pvc-obs-auto-example

Where,

● name indicates the name of the pod to be created.

● app indicates the name of a pod workload.

● image indicates the download address of an image.

● mountPath indicates a mounting path.


./kubectl create -f obs-pod-example.yaml

Step 6 After the pod is created, log in to the CCE console, choose Resource Management> Storage in the navigation pane, and then click the OBS tab to view the bindingrelationship between the workload and PVC.

----End



5.2 Expanding the Disk Capacity of a Node in a CCECluster

Disk Type● System Disk● Node Data Disk (Dedicated for Docker)● Container Disk Space (10 GB)

System Disk

Step 1 Expand the capacity on the EVS console. For details, see Expansion Overview.

Step 2 Restart the node on the ECS console.

Log in to the management console, select the region where the ECS is located,and choose Service List > Computing > Elastic Cloud Server. In the ECS list,locate the target node, and click More > Restart in the Operation column.

----End

Node Data Disk (Dedicated for Docker)

Step 1 Expand the capacity of the Docker disk on the EVS console.

Step 2 Log in to the target node.

Step 3 Run the following commands on the node to add the new disk capacity to theDocker disk:pvresize /dev/vdb lvextend -l+100%FREE -n vgpaas/thinpool

----End

Container Disk Space (10 GB)Log in to the node. Add dm.basesize=15 to the /etc/docker/daemon.json file, asshown in the following figure.

Restart the Docker.

systemctl restart docker

After the container images on the node are deleted, they need to be downloadedagain.



https://support.huaweicloud.com/en-us/usermanual-evs/evs_01_0006.html

https://console.huaweicloud.com/ecm/?locale=en-us#/ecs/manager/vmList

CA UTION

The container disk space is determined by the configuration when the containerimage is downloaded. When deleting an image, you need to clear all layers of theimage for the deletion to take effect.

5.3 Mounting an Object Storage Bucket of a Third-Party Tenant

This section describes how to mount OBS buckets and OBS parallel file systems(preferred) of third-party tenants.

Scenario

The CCE cluster of a SaaS service provider needs to be mounted with the OBSbucket of a third-party tenant, as shown in Figure 5-1.

Figure 5-1 Mounting an OBS bucket of a third-party tenant

1. The third-party tenant authorizes the SaaS service provider to access theOBS buckets or parallel file systems by setting the bucket policy and bucketACL.

2. The SaaS service provider statically imports the OBS buckets and parallelfile systems of the third-party tenant.

3. The SaaS service provider processes the service and writes the processingresult (result file or result data) back to the OBS bucket of the third-partytenant.



Precautions● Only parallel file systems and OBS buckets of third-party tenants in the same

region can be mounted.● Only clusters where the everest add-on of v1.1.11 or later has been installed

(the cluster version must be v1.15 or later) can be mounted with OBS bucketsof third-party tenants.

● The service platform of the SaaS service provider needs to manage thelifecycle of the third-party bucket PVs. When a PVC is deleted separately, thePV is not deleted. Instead, it will be retained. To do so, you need to call thenative Kubernetes APIs to create and delete static PVs.

Authorizing the SaaS Service Provider to Access the OBS BucketsThe following uses an OBS bucket as an example to describe how to set a bucketpolicy and bucket ACL to authorize the SaaS service provider. The configuration foran OBS parallel file system is the same.

Step 1 Log in to the OBS console. In the navigation pane, choose Object Storage.

Step 2 In the bucket list, click a bucket name, and then the Overview page of the bucketis displayed.

Step 3 In the navigation pane on the left, click Permissions to go to the permissionmanagement page.

Step 4 On the Bucket Policies page, click Create Bucket Policy under Custom BucketPolicies, configure the bucket policy as shown in the following figure, and clickOK.

Figure 5-2 Creating a custom bucket policy



● Policy Mode: If this parameter is set to Read and write, the authorized userhas the read and write permissions on the specified object in the bucket.

● Principal: Select Include and Other account, and enter the account ID anduser ID. The bucket policy takes effect for the specified users.

● Resources: Select Include and set the resource name to *. The bucket policytakes effect for all OBS resources.

Step 5 On the Bucket ACLs tab page, click Add. Enter the account ID or account name ofthe authorized user, select Read, Object read, and Write for Access to Bucket,select Read and Write for the Access to ACL, and click Save.

Figure 5-3 Configuring a bucket ACL

----End

Statically Importing OBS Buckets and Parallel File Systems● Static PV of an OBS bucket:

apiVersion: v1kind: PersistentVolumemetadata: name: objbucket #Replace the name with the actual PV name of the bucket. annotations: pv.kubernetes.io/provisioned-by: everest-csi-provisionerspec: accessModes: - ReadWriteMany capacity: storage: 1Gi mountOptions: - default_acl=bucket-owner-full-control #New OBS mounting parameters csi: driver: obs.csi.everest.io fsType: s3fs volumeAttributes: everest.io/obs-volume-type: STANDARD everest.io/region: cn-north-4 #Set it to the ID of the current region. storage.kubernetes.io/csiProvisionerIdentity: everest-csi-provisioner volumeHandle: objbucket #Replace the name with the actual bucket name of the third-party tenant. persistentVolumeReclaimPolicy: Retain #This parameter must be set to Retain to ensure that the bucket will not be deleted when a PV is deleted. storageClassName: csi-obs-mountoption #You can associate a new custom OBS storage class or the built-in csi-obs of the cluster.

– mountOptions: This field contains the new OBS mounting parametersthat allow the bucket owner to have full access to the data in the bucket.This field solves the problem that the bucket owner cannot read the datawritten into a mounted third-party bucket. If the object storage of athird-party tenant is mounted, default_acl must be set to bucket-owner-



full-control. For details about other values of default_acl, see Table 1 inConfiguring an ACL Using Header Fields.

– persistentVolumeReclaimPolicy: When the object storage of a third-party tenant is mounted, this field must be set to Retain. In this way, theOBS bucket will not be deleted when a PV is deleted. The serviceplatform of the SaaS service provider needs to manage the lifecycle ofthe third-party bucket PVs. When a PVC is deleted separately, the PV isnot deleted. Instead, it will be retained. To do so, you need to call thenative Kubernetes APIs to create and delete static PVs.

– storageClassName: You can associate a new custom OBS storage class(click here) or the built-in csi-obs of the cluster.

PVC of a bound OBS bucket:apiVersion: v1kind: PersistentVolumeClaimmetadata: annotations: csi.storage.k8s.io/fstype: s3fs everest.io/obs-volume-type: STANDARD volume.beta.kubernetes.io/storage-provisioner: everest-csi-provisioner: name: objbucketpvc #Replace the name with the actual PVC name of the bucket. namespace: defaultspec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi storageClassName: csi-obs-mountoption #The value must be the same as the storage class associated with the bound PV. volumeName: objbucket #Replace the name with the actual PV name of the bucket to be bound.

● Static PV of an OBS parallel file system:apiVersion: v1kind: PersistentVolumemetadata: name: obsfscheck #Replace the name with the actual PV name of the parallel file system. annotations: pv.kubernetes.io/provisioned-by: everest-csi-provisionerspec: accessModes: - ReadWriteMany capacity: storage: 1Gi mountOptions: - default_acl=bucket-owner-full-control #New OBS mounting parameters csi: driver: obs.csi.everest.io fsType: obsfs volumeAttributes: everest.io/obs-volume-type: STANDARD everest.io/region: cn-north-7 storage.kubernetes.io/csiProvisionerIdentity: everest-csi-provisioner volumeHandle: obsfscheck #Replace the name with the actual name of the parallel file system of the third-party tenant. persistentVolumeReclaimPolicy: Retain #This parameter must be set to Retain to ensure that the bucket will not be deleted when a PV is deleted. storageClassName: csi-obs-mountoption #You can associate a new custom OBS storage class or the built-in csi-obs of the cluster.

– mountOptions: This field contains the new OBS mounting parametersthat allow the bucket owner to have full access to the data in the bucket.This field solves the problem that the bucket owner cannot read the datawritten into a mounted third-party bucket. If the object storage of a



https://support.huaweicloud.com/en-us/devg-obs/obs_06_0043.html

third-party tenant is mounted, default_acl must be set to bucket-owner-full-control. For details about other values of default_acl, see Table 1 inConfiguring an ACL Using Header Fields.

– persistentVolumeReclaimPolicy: When the object storage of a third-party tenant is mounted, this field must be set to Retain. In this way, theOBS bucket will not be deleted when a PV is deleted. The serviceplatform of the SaaS service provider needs to manage the lifecycle ofthe third-party bucket PVs. When a PVC is deleted separately, the PV isnot deleted. Instead, it will be retained. To do so, you need to call thenative Kubernetes APIs to create and delete static PVs.

– storageClassName: You can associate a new custom OBS storage class(click here) or the built-in csi-obs of the cluster.

PVC of a bound OBS parallel file system:apiVersion: v1kind: PersistentVolumeClaimmetadata: annotations: csi.storage.k8s.io/fstype: obsfs everest.io/obs-volume-type: STANDARD volume.beta.kubernetes.io/storage-provisioner: everest-csi-provisioner name: obsfscheckpvc #Replace the name with the actual PVC name of the parallel file system. namespace: defaultspec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi storageClassName: csi-obs-mountoption #The value must be the same as the storage class associated with the bound PV. volumeName: obsfscheck #Replace the name with the actual PV name of the parallel file system.

● (Optional) Creating a custom OBS storage class to associate with a staticPV:apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: csi-obs-mountoptionmountOptions: - default_acl=bucket-owner-full-controlparameters: csi.storage.k8s.io/csi-driver-name: obs.csi.everest.io csi.storage.k8s.io/fstype: s3fs everest.io/obs-volume-type: STANDARDprovisioner: everest-csi-provisionerreclaimPolicy: RetainvolumeBindingMode: Immediate

– csi.storage.k8s.io/fstype: File type. The value can be obsfs or s3fs. If thevalue is s3fs, an OBS bucket is created and mounted using s3fs. If thevalue is obsfs, an OBS parallel file system is created and mounted usingobsfs.

– reclaimPolicy: Reclaim policy of a PV. The value will be set inPV.spec.persistentVolumeReclaimPolicy dynamically created based onthe new PVC associated with the storage class. If the value is Delete, theexternal OBS bucket and the PV will be deleted when the PVC is deleted.If the value is Retain, the PV and external storage are retained when thePVC is deleted. In this case, you need to clear the PV separately. In thescenario where an imported third-party bucket is associated, the storage



https://support.huaweicloud.com/en-us/devg-obs/obs_06_0043.html

class is used only for associating static PVs (with this field set to Retain).Dynamic creation is not involved.



6 Domain Name Resolution

6.1 Configuring Domain Name Resolution for CCEContainers

This section describes how to configure domain name resolution for CCEcontainers.

Service● Create a Service before you create the appropriate back-end workload

(Deployment or ReplicaSet) and any workloads that need to access it. Whenthe Kubernetes starts a container, it provides environment variables that pointto all the Services that are running when the container is started. Forexample, if a Service named foo exists, all containers will obtain the followingvariables when they are initialized.FOO_SERVICE_HOST=<the host the Service is running on> FOO_SERVICE_PORT=<the port the Service is running on>

This requires that any Service that a pod wants to access must be createdbefore the pod is created. Otherwise, environment variables will not takeeffect. This restriction does not apply to DNS.

● For a cluster, an optional add-on is a DNS server (mandatory for CCEclusters). The DNS server monitors the Kubernetes APIs for the new Servicesand creates a set of DNS records for each Service. If DNS is enabledthroughout the cluster, all pods will be able to automatically resolve thenames of Services.

● Do not specify a hostPort for a pod unless necessary. When a pod is bound toa hostPort, the number of locations to which the pod can be scheduled will belimited because each <hostIP, hostPort, protocol> must be unique. If you donot specify hostIP and protocol, Kubernetes uses 0.0.0.0 as the default host IPaddress and TCP as the default protocol.

If you only need to access the port for debugging, you can use apiserver proxies orkubectl port-forward.

If you want to open the pod port on the node, consider using the NodePortService before using hostPort.

Cloud Container EngineBest Practice 6 Domain Name Resolution


● Do not use hostNetwork. The reason is the same as that of using hostPort.● When kube-proxy load balancing is not required, use headless Services

(ClusterIP set to None) for service discovery.

DNSBy default, CCE provides a DNS add-on Service named coredns to automaticallyassign DNS domain names for other Services. If it is running in the cluster, run thefollowing command to check the status:

kubectl get services coredns --namespace=kube-systemNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEkube-dns ClusterIP 10.0.0.10 <none> 53/UDP,53/TCP 8m

If the pod is not running, you can run the describe command to check why thepod is not started. Assume that there is a Service that has a permanent IP addressand a DNS server (coredns cluster add-on) that assigns domain name to the IPaddress. In this way, any pod in the cluster can communicate with the Service. Youcan run another application for testing. Enable a new pod, access the pod, and runthe curl command to check whether the domain name of the Service can becorrectly resolved. In some cases, the curl command cannot be executed due tothe DNS search principles and configuration.

When a pod is created on the CCE console, not all dnsConfig configurations areopened and some default values of the pod domain name resolution parametersare used. You need to know well the default configurations. A typical case isndots. If the number of dots is within the ndots threshold range, the domainname is considered as an internal domain name of the Kubernetes cluster andthe ..svc.cluster.local suffix is added to the domain name.

DNS Search Principles and RulesDNS configuration file: /etc/resolv.conf

nameserver 10.247.x.xsearch default.svc.cluster.local svc.cluster.local cluster.localoptions ndots:3

Parameter description:

● nameserver: domain name resolution server● search: domain name suffix search rule. More search configurations indicate

more matching times for domain name resolution. For example, if threesuffixes are matched, at least six search operations are required because bothIPv4 and IPv6 addresses need to be checked.

● options: domain name resolution option. Multiple KV values are available. Atypical case is ndots. If the number of dots in the domain name to beaccessed exceeds the value of ndots, the domain name is considered as acomplete domain name and is directly parsed. If the number of dots is lessthan the value of ndots, the suffix ..svc.cluster.local will be added.

Parameters in Kubernetes dnsConfig● nameservers: a list of IP addresses that will be used as DNS servers for the

pod. A maximum of three IP addresses can be specified. If pod's dnsPolicy isset to None, the list must contain at least one IP address, otherwise this



property is optional. The servers listed will be combined to the basenameservers generated from the specified DNS policy with duplicateaddresses removed.

● searches: a list of DNS search domains for host name lookup in the pod. Thisproperty is optional. When specified, the provided list will be merged into thebase search domain names generated from the chosen DNS policy. Duplicatedomain names are removed. Kubernetes allows for at most 6 search domains.

● options: an optional list of objects where each object may have a nameproperty (required) and a value property (optional). The contents in thisproperty will be merged to the options generated from the specified DNSpolicy. Duplicate entries are removed.

For details, see DNS for Services and Pods.

Pod DNS PoliciesDNS policies can be set on a per-pod basis. These policies are specified in thednsPolicy field in the YAML file.

● Default: The pod inherits the name resolution configuration from the nodewhere the pod is running.

● ClusterFirst: Any DNS query that does not match the configured clusterdomain suffix (such as "www.kubernetes.io") is forwarded to the upstreamname server inherited from the node. Cluster administrators may have extrastub-domain and upstream DNS servers configured.

● ClusterFirstWithHostNet: For pods running with hostNetwork, set the DNSpolicy to this option.

● None: newly introduced in Kubernetes v1.9 (Beta in v1.10), which allows thepod to ignore the DNS settings from the Kubernetes environment. All DNSsettings are supposed to be provided using the dnsConfig field in the PodSpec. DNS configuration examples are provided as follows:

Scenario 1: Using a custom DNS

The following example allows you to use a custom DNS to resolve the applicationdomain name configuration in pods. After application migration, you do not needto modify the configuration.

apiVersion: v1kind: Podmetadata: namespace: default name: dns-examplespec: containers: - name: test image: nginx dnsPolicy: "None" dnsConfig: nameservers: - 1.2.3.4 searches: - ns1.svc.cluster.local - my.dns.search.suffix options: - name: ndots value: "2" - name: edns0



https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

Scenario 2: Using the Kubernetes DNS add-on CoreDNS

The DNS service of Kubernetes is preferentially used for domain name resolution.If the resolution fails, the DNS service of an external cascading system is used fordomain name resolution.apiVersion: v1kind: Podmetadata: namespace: default name: dns-examplespec: containers: - name: test image: nginx dnsPolicy: ClusterFirst

Scenario 3: Using the public network domain name resolution

This mode applies to the scenario where the domain names in pods are to beaccessed from public networks. In this case, the applications in the pods resolvedomain names from an external DNS.apiVersion: v1kind: Podmetadata: namespace: default name: dns-examplespec: containers: - name: test image: nginx dnsPolicy: Default

Scenario 4: Using hostNetwork

If hostNetwork: true is used to configure the networking in the pod, the networkports of the host machine are exposed to the application running in the pod. Allnetwork ports on the LAN where the host machine is located can be used toaccess the application.apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: nginxspec: template: metadata: labels: app: nginx spec: hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80

If dnsPolicy: ClusterFirstWithHostNet is not added, even if the pod uses the DNSof the host machine by default, other pods in the Kubernetes cluster cannot beaccessed through the Service name in the container.

CoreDNS Configuration1. Configuring the CoreDNS ConfigMap



The default CoreDNS configuration file is as follows:

Corefile: | .:53 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance

Parameter description:

● error: Errors are recorded in stdout.● health: The CoreDNS running status report can be obtained from http://

localhost:8080/health.● kubernetes: The CoreDNS returns a DNS query response based on the IP

addresses of the Kubernetes Service and pod.● prometheus: The measurement standard of CoreDNS can be found in the

metrics in the format of http://localhost:9153/Prometheus. You can obtainmonitoring data in Prometheus format from http://localhost:9153/metrics.

● proxy and forward: Any query that is not in the Kubernetes cluster domain isforwarded to the predefined resolver (/etc/resolv.conf). If the domain namecannot be resolved locally, query the upper-level address. By default, the /etc/resolv.conf configuration of the host machine is used.

● cache: The front-end cache is enabled.● loop: Simple forwarding loops are detected. If a loop is detected, the CoreDNS

process is stopped.● reload: The changed Corefile can be automatically reloaded. After editing the

ConfigMap, wait for two minutes for the modification to take effect.● loadbalance: This is a round-robin DNS load balancer that randomizes the

order of A, AAAA, and MX records in the answer.

2. Configuring an external DNS server

Some services are not in the Kubernetes environment and need to be accessedthrough the DNS. The suffix of the service name is carey.com.

carey:53 { errors cache 30 proxy . 10.150.0.1 }

Complete configuration file:

Corefile: | .:53 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream



fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } carey.com:53 { errors cache 30 proxy . 10.150.0.1 }

Currently, CCE add-on management supports the configuration of stub-domains,which is more flexible and convenient than the direct editing of ConfigMaps. Youdo not need to configure the domain name resolution for pods.

Configuration on the CCE Console

Step 1 Log in to the CCE console. In the navigation pane, choose Add-ons. On the Add-on Instance tab page, click coredns to go to the details page. Click theParameters tab page to view the known parameters and supported parameters ofthe coredns add-on.

Figure 6-1 coredns details page

Step 2 Click Edit on the Parameters tab page to edit the stub domain. That is, configurethe domain name server for the custom domain name. The key is the domainname with the DNS suffix, and the value is one or a group of DNS IP addresses, asshown in Figure 6-2. It is equal to directly editing the ConfigMaps of coredns, butthis is a more efficient way, and is thereby recommended.




Figure 6-2 Editing the stub-domain

----End



7 Charts

7.1 Interconnecting CCE with Helm

OverviewHelm is a package manager for Kubernetes and manages charts. A Helm chart is aseries of YAML files used to encapsulate native Kubernetes applications. Whendeploying an application, you can customize some metadata of the application foreasy application distribution. Application releasers can use Helm to packageapplications, manage application dependencies and application versions, andrelease applications to the software repository. After using Helm, users do notneed to compile complex application deployment files. They can easily search for,install, upgrade, roll back, and uninstall applications on Kubernetes.

The relationship between Helm and Kubernetes is as follows:

● Helm <–> Kubernetes● Apt <–> Ubuntu● Yum <–> CentOS● Pip <–> Python

The following figure shows the solution architecture:

Cloud Container EngineBest Practice 7 Charts


Helm can help application orchestration for Kubernetes:

● Manages, edits, and updates a large number of Kubernetes configuration files.● Deploys a complex Kubernetes application that contains a large number of

configuration files.● Shares and reuses Kubernetes configurations and applications.● Supports multiple environments with parameter-based configuration

templates.● Manages the release of applications, including rolling back the application,

finding differences (using the diff command), and viewing the release history.● Controls phases in a deployment cycle.● Tests and verifies the released version.

PrerequisitesThe created cluster in CCE has been connected to kubectl. For details, seeConnecting to a CCE Cluster Using kubectl or web-terminal.

Procedure● Installing Helm● Installing the Helm Chart● Cloud-based Practice● Reference




● FAQs

Installing HelmInstall Helm on CCE nodes.

You can download Helm of a proper version from here. This section uses Helmv3.3.0 as an example.

Step 1 Download the Helm client on the CCE node.wget https://get.helm.sh/helm-v3.3.0-linux-amd64.tar.gz

Step 2 Decompress the Helm package.tar -xzvf helm-v3.3.0-linux-amd64.tar.gz


Step 3 Copy Helm to the system path, for example, /usr/local/bin/helm.mv linux-amd64/helm /usr/local/bin/helm

Step 4 View the Helm version information.helm versionversion.BuildInfo{Version:"v3.3.0", GitCommit:"e29ce2a54e96cd02ccfce88bee4f58bb6e2a28b6", GitTreeState:"clean", GoVersion:"go1.13.4"}

Step 5 Install the tiller.

NO TE

In Helm v3.0.0 and later versions, tiller is no longer required. Therefore, you do not need toinstall it. Helm reads ~/.kube/config by default to connect to Kubernetes.

----End

Installing the Helm ChartIf the charts provided by CCE do not meet requirements, download a chart andinstall it.

● Method 1: Obtain the required chart in the stable directory on the website,download the chart, and upload it to the node.

a. Download and decompress the obtained chart. Generally, the chart is inZIP format.unzip chart.zipunzip chart.zip

b. Install the Helm chart.helm install chartAfter the installation is complete, log in to the CCE console to view theresources.

● Method 2: Query the chart by running the helm search command and installthe chart.



https://github.com/helm/helm/releases

https://github.com/helm/charts


a. Configure the repository.

helm init --client-only

b. Edit the .helm/repository/repositories.yaml file to change the addressof the chart repository.

vi .helm/repository/repositories.yaml

Modify the following information in bold:apiVersion: v1generated: 2019-10-09T16:06:02.210036264+08:00repositories:- caFile: "" cache: /root/.helm/repository/cache/stable-index.yaml certFile: "" keyFile: "" name: stable password: "" url: http://mirror.azure.cn/kubernetes/charts/ username: ""- caFile: "" cache: /root/.helm/repository/cache/local-index.yaml certFile: "" keyFile: "" name: local password: "" url: http://127.0.0.1:8879/charts username: ""

c. View and install the chart. You can view the downloaded chart inthe .helm/cache/archive directory.

▪ Command for checking the chart repository address:

helm repo list

▪ Command for searching for the chart:

helm search Chart name

▪ Command for installing the chart:

helm install Chart nameThe following is an example of installing the chart:



Cloud-based PracticeThis practice describes how to modify the files in the chart package using thecreated resources. The chart package consists of the chart.yaml, README.md,values.yaml, .helmianore, and templates folders. You can modify the chartpackage by modifying the parameters in the values.yaml folder according to thedeployment.yaml, statefulset.yaml, and service.yaml files in the templatesfolder.

Step 1 Modify the image parameters in the values.yaml file based on the existing imagesand versions in the account.

Step 2 Use an EVS disk. For details, see Using an EVS Disk.

Step 3 Use the ELB. For details, see Using HUAWEI CLOUD ELB.

Step 4 Change the values of the following service parameters in the values.yaml file:● Change the value of key in kubernetes.io/elb.class to union, indicating that

a shared load balancer is used.● Change the value of loadBalancerIP to the address of the specified ELB.

----End

Reference● Helm Charts● Preparing a Chart

FAQs● The following error message is displayed after the helm version command is

run:Client:&version.Version{SemVer:"v3.3.0",GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}E0718 11:46:10.132102 7023 portforward.go:332] an error occurredforwarding 41458 -> 44134: error forwarding port 44134 to podd566b78f997eea6c4b1c0322b34ce8052c6c2001e8edff243647748464cd7919, uid : unableto do port forwarding: socat not found.Error: cannot connect to Tiller

The preceding information is displayed because the socat is not installed. Runthe following command to install the socat:yum install socat -y

● The following error message is displayed after the helm version command isrun (the socat has been installed):





https://github.com/helm/charts


test@local:~/k8s/helm/test$ helm versionClient: &version.Version{SemVer:"v3.3.0", GitCommit:"021cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}Error: cannot connect to Tiller

The Helm chart reads the configuration certificate from the .Kube/config fileto communicate with Kubernetes. The preceding error information isdisplayed because the kubectl configuration is incorrect. In this case,reconnect to kubectl. For details, see Connecting to a CCE Cluster Usingkubectl or web-terminal.

● The storage fails to be created after the cloud storage is connected.This issue may be caused by the annotation field in the created PVC. Changethe chart name and install the chart again.

7.2 Installing nginx-ingress Using a Chart

Scenario

The open-source ingress-controller is used as the load distribution entry of thecluster.

Ingress is a Kubernetes resource object that defines rules for managing externalaccess to the Services in a cluster. The ingress-controller listens on the apiserver todetect addition, deletion, and other operations on Services in the cluster, pushesthe information based on ingress rules to the load balancer (a revert proxy) in realtime, and makes the updates take effect.

NO TE

● The nginx-ingress add-on has been brought offline in the latest version of CCE. Youcan install the nginx-ingress chart from Charts > Sample Charts on the CCE console.For details, see Sample Charts.

● Do not manually modify or delete the load balancer and listener that are automaticallycreated by CCE. Otherwise, the workload will be abnormal. If you have modified ordeleted them by mistake, restart ingress-controller in the background to restore them.

Prerequisites

Before creating a containerized workload, you must have an available cluster. If nocluster is available, create one by following the procedure described in Buying aHybrid Cluster.

Installing the nginx-ingress Chart

Step 1 Log in to the CCE console. In the navigation pane, choose Charts > SampleCharts.

Step 2 On the Sample Charts tab page, you can view all available charts.● Sample charts: provided by open source communities for users to deploy

workloads.You can click a chart to view the chart details, including chart introduction(overview and example), version record (change history of the current chart),and installation record (list of workloads created from the chart by the user).









● Releases: list of workloads that are installed using a chart.

Step 3 On the Sample Charts tab page, click Install Chart under the nginx-ingress chart.In the dialog box displayed, select I have read the above statements and clickOK to go to the installation page.

Step 4 Set the installation parameters listed in Table 7-1. The parameters marked withan asterisk (*) are mandatory.

Figure 7-1 Installing the nginx-ingress Chart

Table 7-1 Parameter description


* Release Name Name of a release, for example, nginx-ingress.

Chart Name Name of the chart that is being installed, for example, nginx-ingress.

* Chart Version Version of the chart.

* Cluster Cluster to which the workload belongs.

* Namespace Namespace to which the new workload belongs. By default,this parameter is set to default.

* WorkloadDeploymentSpecifications

Number of ingress-controller replicas. You can also click AddCustom Specifications to customize the number of ingress-controller replicas based on service requirements.To ensure high availability, select multiple replicas. In thisexample, select three replicas.

Step 5 After the configuration is complete, you can click Install Now or Next.

● Install Now: Confirm the specifications and click Submit. Go to Step 8 toview the workload that is successfully installed using the chart.



● Next: Set the parameters for custom installation as described in Step 6.

Step 6 (Optional) Set parameters for the custom installation.

1. For details about the storage and Service parameters, see Table 7-2.

Table 7-2 Advanced settings for custom installation


Storage 1. If you select Yes, cloud storage is used.2. You can use the default storage allocation or click Edit to

modify the storage subtype and capacity.

Services The default value is Intra-cluster access.– Intra-cluster access: The system automatically allocates a

virtual IP address that can be accessed only by containersin a cluster.

– Intra-VPC access: A workload can be accessed from otherworkloads in the same VPC through a node IP address.

You can also click Edit to change the access configurations.For details, see Network Management.

2. Click Next, confirm the specifications, and click Submit.

Step 7 Return to Releases tab page. The execution status is Installing. Wait until theinstallation is complete.

Step 8 After the installation is successful, the value of Execution Status on the Releasestab page is Installation successful.

Step 9 Click View Access Mode to view parameters such as the access address.

Step 10 Click the name of the release that has been successfully installed to view itsdetails, as shown in the following table.

Table 7-3 Release details

Tab Description

Workload List 1. Running status, type, and number of pods of workloads aredisplayed.

2. Click a workload name to view its details, such as the CPUusage, memory usage, events, and container information.

ReleaseParameters

The configured release parameters are displayed.

----End

7.3 Using Kubeflow and Volcano to Train an AI Model




7.3.1 IntroductionKubernetes has become the de facto standard for cloud native applicationorchestration and management. An increasing number of applications aremigrated to Kubernetes. AI and machine learning inherently involve a largenumber of computing-intensive tasks. Kubernetes is a preferential tool fordevelopers building AI platforms because of its excellent capabilities in resourcemanagement, application orchestration, and O&M monitoring.

Emergence and Constraints of KubeflowBuilding an end-to-end AI computing platform based on Kubernetes is complexand demanding. More than a dozen of phases is required, as shown in thefollowing diagram. Apart from the familiar model training phase, the process alsoincludes data collection, preprocessing, resource management, feature extraction,data verification, model management, model release, and monitoring, as shown inFigure 7-2. If AI algorithm engineers want to run a model training task, they haveto build an entire AI computing platform first. Imagine how time- and labor-consuming that is and how much knowledge and experience it requires.

Figure 7-2 Model training

Kubeflow was released in 2017, which is built on containers and Kubernetes. Itaims to provide data scientists, machine learning engineers, and system O&Mpersonnel with a platform for agile deployment, development, training, release,and management of machine learning services. It leverages the advantages ofcloud native technologies to enable users to quickly and easily deploy, use, andmanage the most popular machine learning software.

Kubeflow 1.0 is now available, providing capabilities in development, building,training, and deployment that cover the entire process of machine learning anddeep learning for enterprise users.

Diagram:



With Kubeflow 1.0, you first develop a model using Jupyter, and then set upcontainers using tools such as Fairing (SDK). Next, you create Kubernetesresources to train the model. After the training is complete, you create and deployservers for inference using KFServing. This is how you use Kubeflow to establish anend-to-end agile process of a machine learning task. This process can be fullyautomated using pipelines, which help achieve DevOps in the AI field.

Kubernetes Pain Points

Does that mean we can now sit back and relax? Not yet. Kubeflow uses thedefault scheduler of Kubernetes, which was initially designed for long services. Itsscheduling capability is inadequate for tasks that involve batch computing andelastic scheduling in AI and big data scenarios. The main constraints are asfollows:

Resource preemption

A TensorFlow job involves two roles: parameter server (ps) and worker. Only whenpods of these two roles run properly at the same time can a TensorFlow job beexecuted normally. However, the default scheduler is insensitive to the roles ofpods in a TensorFlow job. Pods are treated identically and scheduled one by one.This causes problems when there are multiple jobs to schedule and clusterresources are scarce. Each job could end up being allocated with only part of theresources it needs to finish the execution. That is, resources are used up while nojob can be successfully executed. To better illustrate this dilemma, assume thatyou want to run two TensorFlow jobs, namely, TFJob1 and TFJob2. Each of thesejobs has four workers, which means each job requires four GPUs to run. However,your cluster only has four available GPUs in total. In this case, with the defaultscheduler, TFJob1 and TFJob2 could end up being allocated two GPUs each. Theyare waiting each other to finish and release the resources. However, this will nothappen until you manually intervene. The deadlock created in this situation causeresource wastes and low efficiency in job execution.



Lack of affinity-based scheduling

In distributed training, data is frequently exchanged between parameter serversand workers. To ensure higher efficiency, parameter servers and workers of thesame job should be scheduled to the same node for faster transmission using localnetworks. However, the default scheduler is insensitive to the affinity betweenparameter servers and workers of the same job. Pods are randomly scheduledinstead. As shown in the following figure, assume that you want to run twoTensorFlow jobs with each having one ps and two workers. With the defaultscheduler, the scheduling results could be any of the following three situations.However, only result (c) can deliver the highest efficiency. In (c), the ps and theworkers can use the local network to communicate more efficiently and shortenthe training time.

Volcano, a Perfect Batch Scheduling System for Accelerating AI Computing

Volcano is an enhanced batch scheduling system for high-performance computingworkloads running on Kubernetes. It complements Kubernetes in machinelearning, deep learning, HPC, and big data computing scenarios, providingcapabilities such as gang scheduling, computing task queue management, task-topology, and GPU affinity scheduling. In addition, Volcano enhances batch task



creation and lifecycle management, fair-share, binpack, and other Kubernetes-native capabilities. It fully addresses the constraints of Kubeflow in distributedtraining mentioned above.

For more information about Volcano, visit https://github.com/volcano.

Using Volcano in HUAWEI CLOUDThe convergence of Kubeflow and Volcano, two open source projects, greatlysimplifies and accelerates AI computing workloads running on Kubernetes. Thetwo projects have been recognized by an increasing number of players in the fieldand applied in production environments. Volcano is used in HUAWEI CLOUD CCE,Cloud Container Instance (CCI), and Kubernetes-Native Batch Computing Solution.Volcano will continue to iterate with optimized algorithms, enhanced capabilitiessuch as intelligent scheduling, and new inference features such as GPU Share, tofurther improve the efficiency of Kubeflow batch training and inference.

7.3.2 Implementing Typical Distributed AI Training TasksThis section describes how to perform distributed training of a digital imageclassification model using the MNIST dataset based on Kubeflow and Volcano.

Step 1 Log in to the CCE console and create a hybrid cluster. For details, see Buying aHybrid Cluster.

Step 2 Deploy Volcano on the created CCE Cluster.

In the navigation pane on the left, choose Add-ons. On the Add-on Marketplacetab page, click Install Add-on under volcano. In the Basic Information area onthe Install Add-on page, select the cluster and Volcano version, and click Next:Configuration.



https://github.com/volcano




Figure 7-3 Installing the volcano add-on

The volcano add-on has no configuration parameters. Click Install and wait untilthe installation is complete.

Step 3 Log in to the node and check the deployment progress.

Figure 7-4 Checking the deployment progress

Step 4 Deploy the Kubeflow environment.

1. Create a persistent volume (PV).

a. Run the following command to create a PV folder on each worker node:sudo mkdir /mnt/pv{1..4}

b. Manually create four PVs, which occupy 10 Gi, 10 Gi, 20 Gi, and 20 Gi ofdisk space respectively.

2. Install kfctl and set environment variables.

a. Set environment variables as follows:export KF_NAME=<your choice of name for the Kubeflow deployment>export BASE_DIR=<path to a base directory>export KF_DIR=${BASE_DIR}/${KF_NAME} export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml"

b. Install kfctl.tar -xvf kfctl_v1.0.2_<platform>.tar.gzchmod +x kfctlmv kfctl /usr/local/bin/

3. Deploy Kubeflow.cd ${KF_DIR}kfctl apply -V -f ${CONFIG_URI}

Step 5 Deploy the MNIST dataset.

1. Download kubeflow/examples to the local host and select an operationguide based on the environment.git clone https://github.com/kubeflow/examples.git

2. Install and start Jupyter Notebook.pip3 install jupyter notebookjupyter notebook --allow-root



3. Configure an SSH tunnel on PuTTY and remotely connect to the notebook.4. After the connection is successful, enter localhost:8000 in the address box of

a browser to log in to the notebook.

5. Create a distributed training job as prompted by Jupyter. Set the value ofschedulerName to volcano to enable the Volcano scheduler.kind: TFJobmetadata: name: {train_name} spec: schedulerName: volcano tfReplicaSpecs: Ps: replicas: {num_ps} template: metadata: annotations: sidecar.istio.io/inject: "false" spec: serviceAccount: default-editor containers: - name: tensorflow command: ... env: ... image: {image} workingDir: /opt restartPolicy: OnFailure Worker: replicas: 1 template: metadata: annotations: sidecar.istio.io/inject: "false" spec: serviceAccount: default-editor containers: - name: tensorflow command: ... env: ... image: {image} workingDir: /opt restartPolicy: OnFailure



Step 6 Submit the job and start the training.kubectl apply -f mnist.yaml

After the training job is complete, you can query the training results on theKubeflow UI. This is how you run a simple distributed training task using Kubeflowand Volcano. Kubeflow simplifies TensorFlow job configuration. Volcano, withsimply one more line of configuration, saves you significant time and effort inlarge-scale distributed training by providing capabilities such as gang schedulingand task topology to eliminate deadlocks and achieve affinity scheduling.

----End



8 Permissions

8.1 Configuring kubeconfig to Implement RefinedManagement on Cluster Resources

ScenarioBy default, the kubeconfig file provided by CCE for users has permissions bound tothe cluster-admin role, which are equivalent to the permissions of user root. It isdifficult to implement refined management on users with such permissions.

PurposeCluster resources are managed in a refined manner so that specific users haveonly certain permissions (such as adding, querying, and modifying resources).

NoteEnsure that kubectl is available on your host. If not, download it from here(corresponding to the cluster version or the latest version).

Configuration MethodNO TE

In the following example, only pods and Deployments in the test space can be viewed andadded, and they cannot be deleted.

Step 1 Set the service account name to my-sa and namespace to test.

kubectl create sa my-sa -n test

Step 2 Configure the role table and assign operation permissions to different resources.

Cloud Container EngineBest Practice 8 Permissions


https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/README.md

vi role-test.yamlapiVersion: rbac.authorization.k8s.io/v1kind: Rolemetadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: myrole namespace: testrules:- apiGroups: - "" resources: - pods verbs: - get - list - watch- apiGroups: - apps resources: - pods - deployments verbs: - get - list - watch - create

kubectl create -f role-test.yaml

Step 3 Create a RoleBinding and bind the service account to the role so that the user canobtain the corresponding permissions.

vi myrolebinding.yamlapiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata: name: myrolebinding namespace: testroleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: myrolesubjects:- kind: ServiceAccount name: my-sa namespace: test

kubectl create -f myrolebinding.yaml

The user information is configured. Now perform Step 4 to Step 6 to write theuser information to the configuration file.

Step 4 Configure the cluster information.

1. Use the sa name my-sa to obtain the secret corresponding to the sa. In thefollowing example, my-sa-token-z4967 in the first column is the secret name.



kubectl get secret -n test |grep my-sa

2. Decrypt the ca.crt file in the ecret and export it.

kubectl get secret my-sa-token-5gpl4 -n test -oyaml |grep ca.crt:|awk '{print $2}' |base64 -d > /home/ca.crt

3. Set the cluster access mode. test-arm indicates the cluster to be accessed,10.0.1.100 indicates the IP address of the API server in the cluster (for detailsabout how to obtain the IP address, see Figure 8-1), and /home/test.configindicates the path for storing the configuration file.– If the internal API server address is used, run the following command:

kubectl config set-cluster test-arm --server=https://10.0.1.100:5443 --certificate-authority=/home/ca.crt --embed-certs=true --kubeconfig=/home/test.config

– If the public API server address is used, run the following command:kubectl config set-cluster test-arm --server=https://10.0.1.100:5443 --kubeconfig=/home/test.config --insecure-skip-tls-verify=true

NO TE

If you perform operations on a node in the cluster or the node that uses theconfiguration is a cluster node, do not set the path of kubeconfig to /root/.kube/config.

The cluster API server address is an intranet API server address. After an EIP isbound to the cluster, the cluster API server address can also be a public API serveraddress. You can obtain the API server address on the cluster details page on theCCE console.



Figure 8-1 Obtaining the internal or public API server address

Step 5 Configure the cluster authentication information.

1. Obtain the cluster token. (If the token is obtained in GET mode, you need torun based64 -d to decode the token.)

token=$(kubectl describe secret my-sa-token-5gpl4 -n test | awk '/token:/{print$2}')

2. Set the cluster user ui-admin.

kubectl config set-credentials ui-admin --token=$token --kubeconfig=/home/test.config

Step 6 Configure the context information for cluster authentication. ui-admin@test isthe context name.

kubectl config set-context ui-admin@test --cluster=test-arm --user=ui-admin --kubeconfig=/home/test.config

Step 7 Set the context. For details about how to use the context, see PermissionsVerification.

kubectl config use-context ui-admin@test --kubeconfig=/home/test.config



NO TE

If you want to assign other users the above permissions to perform operations on thecluster, provide the generated configuration file /home/test.config to the user afterperforming step Step 6. The user must ensure that the host can access the API serveraddress of the cluster. When performing step Step 7 on the host and using kubectl, the usermust set the kubeconfig parameter to the path of the configuration file.

----End

Permissions Verification1. Pods in the test namespace cannot access pods in other namespaces.

kubectl get pod -n test --kubeconfig=/home/test.config

2. Pods in the test namespace cannot be deleted.

Further ReadingsFor more information about users and identity authentication in Kubernetes, seeAuthenticating.



https://kubernetes.io/docs/reference/access-authn-authz/authentication/

9 API&kubectl

9.1 Connecting to Multiple Clusters Using kubectl

ScenarioUse the same kubectl client to switch between clusters by changing the user.

Figure 9-1 Using kubectl to connect to multiple clusters

Prerequisites● The ECS where kubectl is deployed can access the virtual IP addresses and

port 5443 of clusters A and B by using the curl command.● The following configuration is for reference only. For convenience, a node in

cluster A is used as the client (the node is bound with an EIP).● To facilitate access, a public IP address (for example, 1.2.3.4) is bound to the

VIP of cluster B. If cluster A and cluster B are in the same VPC, you do notneed to perform this operation.

Cloud Container EngineBest Practice 9 API&kubectl


Configuring kubectl to Access Cluster AMethod 1: Follow the instruction provided in Connecting to a CCE Cluster Usingkubectl or web-terminal.

Method 2: Call the API.

Perform the following operations on the node:

token=`curl -X POST -H 'Content-Type:application/json' -d '{"auth":{"identity":{"methods":["password"],"password":{"user":{"name": "Name of the IAM user","password":"Password of the IAM user","domain":{"name":"Tenant account"}}}},"scope":{"project":{"name":"cn-north-4"}}}}' -i https://iam.cn-north-4.myhuaweicloud.com/v3/auth/tokens -k -s |grep X-Subject-Token | awk -F ':' '{print $2}'` mkdir $HOME/.kube/config curl -H "X-Auth-Token:$token" -H 'Content-Type: application/json' -X GET -k -v https://cce.cn-north-4.myhuaweicloud.com/api/v3/projects/Project ID/clusters/Cluster ID/clustercert > $HOME/.kube/config

The configuration of access to the cluster A is complete.

Configuring kubectl to Access Cluster BTo access cluster B, you need to obtain the cluster address and the authenticationinformation.

The procedure is as follows:

Step 1 Input the information of cluster B.kubectl config set-cluster cluster-k8s --server=https://1.2.3.4:5443 --insecure-skip-tls-verify=true

--insecure-skip-tls-verify=true must be contained, which is used to ignore theclient certificate verification.

Step 2 Input the authentication information for accessing cluster B.

Perform the following operations on cluster B:

Method 1: Upload the certificate of cluster B to the client. Perform operations onthe client and add user details to the configuration file.

kubectl config set-credentials ui-admin --client-certificate=client.crt --embed-certs=true

Method 2: Obtain the authentication token from cluster B.

1. Create a user named sa.kubectl create sa my-sa

2. Assign permissions to user sa.kubectl create clusterrolebinding myrolebinding --serviceaccount=default:my-sa --clusterrole=cluster-admin

3. Obtain the user token.kubectl describe secret my-sa-token-xxx | awk '/token:/{print $2}' > token

Pass the obtained token to kubectl.4. On kubectl, add the detailed user information to the configuration file.

kubectl config set-credentials ui-admin --token=$token

Step 3 Add the context details of cluster B to the configuration file.kubectl config set-context ui-admin@cyd --cluster=cluster-k8s --user=ui-admin

On kubectl:

Run kubectl config use-context internal to switch to cluster A.

Run kubectl config use-context ui-admin@cyd to switch to cluster B.





Step 4 Verify the configurations.

Troubleshooting:

If the X509 error is reported after the configuration, the possible cause is that the--insecure-skip-tls-verify=true parameter is not input.

In this case, add this parameter, for example, kubectl get pod --insecure-skip-tls-verify=true.

----End

9.2 Using kubectl to Modify hostAliases to ConfigureContainer Parameters

ScenarioIf DNS or other related settings are inappropriate, you can use hostAliases tooverwrite the resolution of the host name at the pod level when adding entries tothe /etc/hosts file of the pod. When you use hostAliases, the file is managed bykubectl and can be rewritten during the pod creation and restart.

PrerequisitesCluster nodes have been connected to kubectl. For details, see Connecting to aKubernetes Cluster Using kubectl or web-terminal.

Procedure

Step 1 Log in to the ECS server on which kubectl has been configured.

Step 2 Create the hostaliases-pod.yaml file.

vi hostaliases-pod.yaml

The field in bold in the YAML file indicates the image name and tag. You canreplace the example value as required.

apiVersion: v1kind: Podmetadata: name: hostaliases-podspec: hostAliases: - ip: 127.0.0.1 hostnames:





- foo.local - bar.local - ip: 10.1.2.3 hostnames: - foo.remote - bar.remote containers: - name: cat-hosts image: tomcat:9-jre11-slim lifecycle: postStart: exec: command: - cat - /etc/hosts

Table 9-1 pod field description

Parameter Mandatory/Optional

Description

apiVersion Mandatory API version number

kind Mandatory Type of the object to be created

metadata Mandatory Metadata definition of a resourceobject

name Mandatory Name of a pod

spec Mandatory Detailed description of the pod. Fordetails, see Table 9-2.

Table 9-2 spec field description


Description

hostAliases Mandatory Host alias

containers Mandatory For details, see Table 9-3.

Table 9-3 containers field description


Description

name Mandatory Container name

image Mandatory Container image name

lifecycle Optional Lifecycle


kubectl create –f hostaliases-pod.yaml



If information similar to the following is displayed, the pod is created.

pod/hostaliases-pod created

Step 4 Query the pod status.

kubectl get pod hostaliases-pod

If the pod is in the Running state, the pod is successfully created.

NAME READY STATUS RESTARTS AGEhostaliases-pod 1/1 Running 0 16m

----End



10 Monitoring

10.1 Connecting a CCE Cluster to Heapster forMonitoring

PrerequisitesThe dashboard add-on has been installed and connected to the cluster usingkubectl.

Procedure

Step 1 Create influxdb.yaml, heapster.yaml, and heapster-rbac.yaml resource files.

1. Create an influxdb.yaml file.vi influxdb.yamlkubectl create -f influxdb.yamlModify the image path in the influxdb.yaml file. Official images aresupported. You can also download the influxdb.yaml file from https://github.com/kubernetes-retired/heapster/blob/master/deploy/kube-config/influxdb/influxdb.yaml.apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: monitoring-influxdb namespace: kube-systemspec: replicas: 1 template: metadata: labels: task: monitoring k8s-app: influxdb spec: containers: - name: influxdb image: k8s.gcr.io/heapster-influxdb-amd64:v1.5.2 volumeMounts: - mountPath: /data name: influxdb-storage

Cloud Container EngineBest Practice 10 Monitoring


https://github.com/kubernetes-retired/heapster/blob/master/deploy/kube-config/influxdb/influxdb.yaml



volumes: - name: influxdb-storage emptyDir: {}---apiVersion: v1kind: Servicemetadata: labels: task: monitoring # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) # If you are NOT using this as an addon, you should comment out this line. kubernetes.io/cluster-service: 'true' kubernetes.io/name: monitoring-influxdb name: monitoring-influxdb namespace: kube-systemspec: ports: - port: 8086 targetPort: 8086 selector: k8s-app: influxdb

2. Create a heapster.yaml file.vi heapster.yamlkubectl create -f heapster.yamlModify the image path in the heapster.yaml file. Official images aresupported. You can also download the heapster.yaml file from https://github.com/kubernetes-retired/heapster/blob/master/deploy/kube-config/rbac/heapster-rbac.yaml.apiVersion: v1kind: ServiceAccountmetadata: name: heapster namespace: kube-system---apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: heapster namespace: kube-systemspec: replicas: 1 template: metadata: labels: task: monitoring k8s-app: heapster spec: serviceAccountName: heapster containers: - name: heapster image: k8s.gcr.io/heapster-amd64:v1.5.4 imagePullPolicy: IfNotPresent command: - /heapster - --source=kubernetes:https://kubernetes.default - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086---apiVersion: v1kind: Servicemetadata: labels: task: monitoring # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) # If you are NOT using this as an addon, you should comment out this line.



https://github.com/kubernetes-retired/heapster/blob/master/deploy/kube-config/rbac/heapster-rbac.yaml



kubernetes.io/cluster-service: 'true' kubernetes.io/name: Heapster name: heapster namespace: kube-systemspec: ports: - port: 80 targetPort: 8082 selector: k8s-app: heapster

3. Create a heapster.yaml file.vi heapster.yamlkubectl create -f heapster.yamlYou can also download the heapster.yaml file from https://github.com/kubernetes-retired/heapster/blob/master/deploy/kube-config/influxdb/heapster.yaml.kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1beta1metadata: name: heapsterroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:heapstersubjects:- kind: ServiceAccount name: heapster namespace: kube-system

Step 2 Add the heapster permission to the ClusterRole resource of the dashboard.

kubectl edit ClusterRole role-dashboard

Add the following information to the end of the rule field:# heapster related access- apiGroups: [""] resources: ["services"] resourceNames: ["heapster"] verbs: ["proxy"]- apiGroups: [""] resources: ["services/proxy"] resourceNames: ["heapster", "http:heapster:", "https:heapster:"] verbs: ["get"]

After the information is added, information similar to the following is displayed:



https://github.com/kubernetes-retired/heapster/blob/master/deploy/kube-config/influxdb/heapster.yaml



Step 3 Add startup parameters to the deploy resource of the dashboard.

kubectl edit deploy kubernetes-dashboard -n kube-system

Add the following information to the end of the spec.spec.containers field:args: - --heapster-host=http://heapster.kube-system

After the information is added, information similar to the following is displayed:

Step 4 Access kubernetes-dashboard at https://EIP:Port.

----End



11 Migrating Applications from a Third-Party Cloud Cluster to HUAWEI CLOUD CCE

11.1 PreparationThis section takes the WordPress application as an example to describe how tomigrate an application from Alibaba Cloud ACK to HUAWEI CLOUD CCE. Assumethat you have deployed the WordPress application on Alibaba Cloud and createdyour own blog.

PrecautionsAlibaba Cloud ACK differs from HUAWEI CLOUD CCE in certain functions. Evaluatethe differences before migrating your applications.

PlanningCluster Planning

CCE allows you to create the following types of clusters to meet your servicerequirements:

● Hybrid cluster (recommended): supports hybrid deployment of VMs andbare-metal servers (BMSs), and heterogeneous nodes such as GPU and NPUnodes. Hybrid clusters provide a comprehensive, secure, and stable containerruntime environment based on a high-performance network model.

● Kunpeng cluster: manages only Kunpeng (Arm-based) VM nodes. Kunpengclusters provide a secure and stable container runtime environment for pureArm-based VM scenarios.

For the WordPress application in this example, a hybrid cluster is recommended.

Network planning

CCE supports the following two network models:

● Tunnel network: The container network is an overlay tunnel network on topof a VPC network and uses the VXLAN technology. This network model is


11 Migrating Applications from a Third-Party CloudCluster to HUAWEI CLOUD CCE


applicable when there is no high requirements on performance. VXLANencapsulates Ethernet packets as UDP packets for tunnel transmission.Though at some cost of performance, the tunnel encapsulation enableshigher interoperability and compatibility with advanced features (such asnetwork policy-based isolation), meeting the requirements of mostapplications.

● VPC network: The container network uses VPC routing to integrate with theunderlying network. This network model is applicable to performance-intensive scenarios. The maximum number of nodes allowed in a clusterdepends on the route quota in a VPC network. Each node is assigned a CIDRblock of a fixed size. VPC networks are free from tunnel encapsulationoverhead and outperform container tunnel networks. In addition, as VPCrouting includes routes to node IP addresses and the container CIDR block,container pods in the cluster can be directly accessed from outside the cluster.

For the WordPress application in this example, the VPC network is recommended.

Storage Planning

CCE supports the following storage modes:

● EVS volumes: Mount an EVS volume to a container path. When containersare migrated, the attached EVS volumes are migrated accordingly. Thisstorage mode is suitable for data that needs to be permanently stored.

● SFS volumes: Create SFS volumes and mount them to a container path. Thefile system volumes created by the underlying SFS service can also be used.SFS volumes are applicable to persistent storage for frequent read/write inmultiple workload scenarios, including media processing, contentmanagement, big data analysis, and workload analysis.

● OBS volumes: Create OBS volumes and mount them to a container path. OBSvolumes are applicable to scenarios such as cloud workload, data analysis,content analysis, and hotspot objects.

● SFS Turbo volumes: Create SFS Turbo volumes and mount them to acontainer path. SFS Turbo volumes are fast, on-demand, and scalable, whichmakes them suitable for DevOps, containerized microservices, and enterpriseoffice applications.

For the WordPress application in this example, EVS volumes are recommended.

11.2 Migration Scheme OverviewThis document briefly describes how to smoothly migrate an application from anAlibaba Cloud ACK cluster to a HUAWEI CLOUD CCE cluster in six steps withoutinterrupting the service.




Migration Scheme




Procedure

11.3 Creating and Configuring a CCE Hybrid ClusterO&M personnel create a HUAWEI CLOUD CCE cluster and configure relatedresources. For details, see Buying a Hybrid Cluster.

Table 11-1 compares the features provided by ACK and CCE and givesconfiguration recommendations on CCE resources involved in the migration.

Table 11-1 Comparison between ACK and CCE features

Feature

Specifications Alibaba Cloud ACKConfigurations

Recommended HUAWEICLOUD CCE Configurations

Cluster

Cluster type Elastic bare metal(X-Dragon) cluster

Hybrid cluster (hybriddeployment of VMs and BMSs,and hybrid deployment ofheterogeneous nodes such asGPU nodes and NPU nodes)

Kubernetesversion

1.14 1.15

1.16 1.17





Feature



Network Flannel (networkpolicy notsupported)

Choose the network type basedon the live environment:Tunnel network (network policysupported; ENI not supported)VPC network (network policynot supported; ENI supported)

Terway (networkpolicy and ENIsupported)

Storage add-ontype

Flexvolume everest add-on

CSI

SNATconfiguration

Enabled After a CCE cluster is created,manually purchase andconfigure a HUAWEI CLOUDNAT Gateway.

Accessing theAPI server overa publicnetwork

Enabled After a CCE cluster is created,go to the cluster details page.On the Kubectl tab page, bindan elastic IP address to thecluster.

kube-proxymode

iptables iptables

IPVS ipvs

CPU policy Static Enabled

None Disabled

Ingress nginx-ingress By default, CCE clusters use ELBto provide layer-7 access.If you want to use nginx toprovide layer-7 access, create aCCE cluster and choose Charts> Sample Charts, and installthe nginx-ingress chart.

Monitoringadd-ons

UsingCloudMonitor tomonitor ECSinstances

By default, CCE clustersinterconnect with HUAWEICLOUD AOM.After a node is created, you canview the monitoring details inthe Host Monitoring page ofAOM.

Using Prometheusto monitor services

Installing the prometheus add-on




Feature



Log services Using the logservice

By default, CCE clustersinterconnect with HUAWEICLOUD AOM.After a cluster is created, addlog policies to containers.

Installing node-problem-detectoreand creating anEvent Center

Installing the npd add-on

Node OS type CentOS CentOS

Aliyun Linux EulerOS and CentOS

Disk type Ultra cloud disk High I/O

SSD cloud disk Ultra-high I/O

Specifications GPUcomputing/NPUcomputing

GPU-accelerated/AI-accelerated

ComputeOptimized Type c6

General computing-plus C6

General PurposeType g6

General computing S6

Memory OptimizedType r6

Memory-optimized M6

Enhanced MemoryOptimized Type r4e

Large-memory E3

Big Data Type d1ne Disk-intensive D3

Local SSD Type i2 Ultra-high I/O I3

11.4 Migrating Data

11.4.1 Migrating Databases and Storage

Migrating DatabasesO&M personnel or development personnel migrate databases using DataReplication Service (DRS). For details, see Migrating Databases Across CloudPlatforms.




https://support.huaweicloud.com/en-us/bestpractice-drs/drs_overview.html

https://support.huaweicloud.com/en-us/bestpractice-drs/drs_overview.html

Migrating StorageO&M personnel or development personnel migrate data in object storage usingObject Storage Migration Service (OMS). For details on OMS, see Object StorageMigration Service.

NO TE

Currently, you can use OMS to migrate object storage data from Amazon Web Services(AWS), Alibaba Cloud, Microsoft Azure, Baidu Cloud, Kingsoft Cloud, QingCloud, QiniuCloud, and Tencent Cloud to HUAWEI CLOUD OBS.

● Create a bucket on OBS. For details, see Creating a Bucket.● Create a migration task on OMS. For details, see Creating an Object Storage

Migration Task.

11.4.2 Migrating Container ImagesStep 1 Export the container images used in ACK clusters.

1. Pull the images to the client by referring to the operation guide of AlibabaCloud Container Registry (ACR).

2. Run the docker save command on the client to save the images as files anddownload the files to the local host.

docker save -o wordpress-4.tar.gz wordpress:4 docker save -o mysql-5.6.tar.gz mysql:5.6

Step 2 Upload the image files to HUAWEI CLOUD SWR.

1. Log in to the SWR console.2. Create an organization, for example, ack2cce.3. In the navigation pane, choose My Images, and upload the images using the

SWR console.4. Record and save the addresses of the uploaded images.

----End

11.5 Migrating the Application

11.5.1 Preparing Object Storage and VeleroO&M or development personnel migrate Kubernetes objects using the Velero tool.

Preparing Object Storage MinIOMinIO official website: https://docs.min.io/

Prepare the object storage and save its AK/SK.

Step 1 Install the MinIO.

MinIO is a high performance,distributed,Kubernetes Native Object Storage.

# Binary installationmkdir /opt/minio




https://support.huaweicloud.com/en-us/productdesc-oms/en-us_topic_0045916963.html

https://support.huaweicloud.com/en-us/productdesc-oms/en-us_topic_0045916963.html

https://support.huaweicloud.com/en-us/productdesc-obs/en-us_topic_0045853681.html

https://support.huaweicloud.com/en-us/usermanual-obs/en-us_topic_0045853662.html

https://support.huaweicloud.com/en-us/qs-oms/en-us_topic_0045916962.html

https://support.huaweicloud.com/en-us/qs-oms/en-us_topic_0045916962.html

https://docs.min.io/

mkdir /opt/miniodata cd /opt/minio wget https://dl.minio.io/server/minio/release/linux-amd64/minio chmod +x minio export MINIO_ACCESS_KEY=minio export MINIO_SECRET_KEY=minio123 ./minio server /opt/miniodata/ & Enter http://{EIP of the node where MinIO is deployed}:9000 in the address box of a browser. Note that the corresponding ports on the firewall and security group must be enabled.

# Installing kubectl in containers# To release the MinIO service as a service that can be accessed from outside the cluster, change the service type in 00-minio-deployment.yaml to NodePort or LoadBalancer.kubectl apply -f ./velero-v1.4.0-linux-amd64/examples/minio/00-minio-deployment.yaml

Step 2 Create a bucket, which will be used in the migration.Open the web page of the MinIO service.Use MINIO_ACCESS_KEY/MINIO_SECRET_KEY to log in to the MinIO service. In this example, use minio/minio123.Click Create bucket above +. In this example, create a bucket named velero.

----End

Preparing Velero

Velero official website: https://velero.io/docs/v1.4/contributions/minio/

Velero is an open source tool to safely back up, restore, perform disaster recovery,and migrate Kubernetes cluster resources and persistent volumes.

Perform the following operations on the ACK and CCE nodes that can run kubectlcommands:

Step 1 Download the migration tool Velero.Download the latest stable version from https://github.com/heptio/velero/releases.This document uses velero-v1.4.0-linux-amd64.tar.gz as an example.

Step 2 Install the Velero client.mkdir /opt/ack2cce cd /opt/ack2cce tar -xvf velero-v1.4.0-linux-amd64.tar.gz -C /opt/ack2cce cp /opt/ack2cce/velero-v1.4.0-linux-amd64/velero /usr/local/bin

Step 3 Install the Velero server.cd /opt/ack2cce # Prepare the MinIO authentication file. The AK/SK must be correct.vi credentials-velero

[default] aws_access_key_id = minio aws_secret_access_key = minio123

# Install the Velero server. Note that s3Url must be set to the correct MinIO address.velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.0.0 \ --bucket velero \ --secret-file ./credentials-velero \ --use-restic \ --use-volume-snapshots=false \ --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://121.36.101.78:9000

----End




https://velero.io/docs/v1.4/contributions/minio/

11.5.2 Backing Up Kubernetes Objects of the ACK ClusterStep 1 If you need to back up a WordPress application with PV data, add an annotation

to the corresponding pod. If you do not need to back up the PV data, skip thisstep.# kubectl -n YOUR_POD_NAMESPACE annotate pod/YOUR_POD_NAME backup.velero.io/backup-volumes=YOUR_VOLUME_NAME_1,YOUR_VOLUME_NAME_2,...

[root@iZbp1cqobeh1iyyf7qgvvzZ ack2cce]# kubectl get pod -n wordpressNAME READY STATUSRESTARTS AGE wordpress-67796d86b5-f9bfm 1/1 Running 1 39m wordpress-mysql-645b796d8d-6k8wh 1/1 Running 0 38m

[root@iZbp1cqobeh1iyyf7qgvvzZ ack2cce]# kubectl -n wordpress annotate pod/wordpress-67796d86b5-f9bfm backup.velero.io/backup-volumes=wordpress-pvc pod/wordpress-67796d86b5-f9bfm annotated [root@iZbp1cqobeh1iyyf7qgvvzZ ack2cce]# kubectl -n wordpress annotate pod/wordpress-mysql-645b796d8d-6k8wh backup.velero.io/backup-volumes=wordpress-mysql-pvc pod/wordpress-mysql-645b796d8d-6k8wh annotated

Step 2 Execute the backup task.[root@iZbp1cqobeh1iyyf7qgvvzZ ack2cce]# velero backup create wordpress-ack-backup --include-namespaces wordpress Backup request "wordpress-ack-backup" submitted successfully. Run `velero backup describe wordpress-ack-backup` or `velero backup logs wordpress-ack-backup` for more details.

Step 3 Check whether the backup task is successful.[root@iZbp1cqobeh1iyyf7qgvvzZ ack2cce]# velero backup get NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR wordpress-ack-backup InProgress 2020-07-07 20:31:19 +0800 CST 29d default<none> [root@iZbp1cqobeh1iyyf7qgvvzZ ack2cce]# velero backup get NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR wordpress-ack-backup Completed 2020-07-07 20:31:19 +0800 CST 29d default<none>

----End

11.5.3 Restoring Kubernetes Objects in the Created CCECluster

Creating a StorageClassIn this example, the WordPress application uses Alibaba Cloud SSD persistent datavolumes, which need to be replaced with HUAWEI CLOUD SSDs.

The StorageClass used in this example is alicloud-disk-ssd. Create an SC with thesame name and use HUAWEI CLOUD SSDs as backend storage media. Set thisparameter based on the application to migrate.

[root@ccenode-roprr hujun]# cat cce-sc-csidisk-ack.yaml allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: alicloud-disk-ssd selfLink: /apis/storage.k8s.io/v1/storageclasses/csi-disk parameters: csi.storage.k8s.io/csi-driver-name: disk.csi.everest.io csi.storage.k8s.io/fstype: ext4 everest.io/disk-volume-type: SSD everest.io/passthrough: "true" provisioner: everest-csi-provisioner




reclaimPolicy: Delete volumeBindingMode: Immediate

[root@ccenode-roprr hujun]# kubectl create -f cce-sc-csidisk-ack.yaml

Restoring the Application[root@ccenode-roprr hujun]# velero restore create --from-backup wordpress-ack-backup Restore request "wordpress-ack-backup-20200707212519" submitted successfully. Run `velero restore describe wordpress-ack-backup-20200707212519` or `velero restore logs wordpress-ack-backup-20200707212519` for more details

[root@ccenode-roprr hujun]# velero restore get NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR wordpress-ack-backup-20200708112940 wordpress-ack-backup Completed 0 02020-07-08 11:29:42 +0800 CST <none>

Check the running status of the WordPress application. Make adaptation if issuessuch as image pulling failures and service access failures occur.

11.5.4 Update and AdaptationApplication configuration items include image address, Service, and storage diskmounting. Update and make adaptation based on the live environment.

In this example, update the WordPress service.

Step 1 On the ELB console, buy an elastic load balancer.

Step 2 Record the ELB ID and the corresponding subnet ID.

Step 3 Log in to the CCE console, choose Resource Management > Network and findthe WordPress service. Add the following annotations to its YAML file:annotations: kubernetes.io/elb.class: union kubernetes.io/elb.id: 9d06a39d-12e1-4ada-835d-c204397498a3 kubernetes.io/elb.subnet-id: f86ba71c-beb2-46e1-8910-39c8a7d4bb36 kubernetes.io/session-affinity-mode: SOURCE_IP

----End

If the image of your application fails to be pulled, edit the YAML file on theworkload page of the CCE console. Set the image address to the repositoryaddress obtained from the SWR console.

11.5.5 Debugging and Starting the ApplicationDebug and access the application to check data.

Step 1 Log in to the CCE console. In the navigation pane, choose ResourceManagement > Network. Click the EIP next to the WordPress service.

Figure 11-1 Obtaining the access address





Step 2 If the access is normal, and the migration is successful.

Figure 11-2 WordPress welcome page

----End

11.6 Others

Service Verification

Testing personnel check the functions of the new cluster without interrupting thelive traffic.

● Configure a test domain name.

● Test service functions.

● Check O&M functions, such as log monitoring and alarm reporting.

Switching Live Traffic to the CCE Cluster

O&M switch DNS to direct live traffic to the CCE cluster.

● DNS traffic switching: Adjust the DNS configuration to switch traffic.

● Client traffic switching: Upgrade the client code or update the configurationto switch traffic.

Bringing the ACK Cluster Offline

After confirming that the service on the CCE cluster is normal, bring the ACKcluster offline and delete the backup files.

● Verify that the service on the CCE cluster is running properly.

● Bring the ACK cluster offline.

● Delete backup files.




12 Containerizing an EnterpriseApplication (ERP)

12.1 OverviewThis chapter provides CCE best practices to walk you through the applicationcontainerization.

What Is a Container?

A container is a lightweight high-performance resource isolation mechanismimplemented based on the Linux kernel. It is a built-in capability of the operatingsystem (OS) kernel.

CCE is an enterprise-class container service based on open-source Docker andKubernetes. It is a high-performance and high-reliability service through whichenterprises can manage containerized applications. CCE supports nativeKubernetes applications and tools, allowing you to easily set up a containerruntime on the cloud.

Why Is a Container Preferred?● More efficient use of system resources

A container does not require extra costs such as fees for hardwarevirtualization and those for running a complete OS. Therefore, a container hashigher resource usage. Compared with a VM with the same configurations, acontainer can run more applications.

● Faster startupA container directly runs on the host kernel and does not need to start acomplete OS. Therefore, a container can be started within seconds or evenmilliseconds, greatly saving the development, testing, and deployment time.

● Consistent runtime environmentA container image provides a complete runtime environment to ensureenvironment consistency. In this case, problems (for example, some code runsproperly on machine A but fails to run on machine B) will not occur.

Cloud Container EngineBest Practice 12 Containerizing an Enterprise Application (ERP)


https://www.huaweicloud.com/en-us/product/cce.html

● Easier application migration, maintenance, and scalingA consistent runtime environment makes application migration easier. Inaddition, the in-use storage and image technologies facilitate the reuse ofrepeated applications and simplifies the expansion of images based on baseimages.

Containerization ModesThe following modes are available for containerizing applications:

● Mode 1: Containerize a single application as a whole. Application code andarchitecture remain unchanged.

● Mode 2: Separate the components that are frequently upgraded or have highrequirements on auto scaling from an application, and then containerize thesecomponents.

● Mode 3: Transform an application to microservices and then containerize themicroservices one by one.

Table 12-1 lists the advantages and disadvantages of the three modes.

Table 12-1 Containerization modes

Containerization Mode Advantage Disadvantage

Mode 1: Containerize asingle application as awhole.

● Zero modification onservices: Theapplication architectureand code require nochange.

● The deployment andupgrade efficiency isimproved. Applicationscan be packed ascontainer images toensure applicationenvironmentconsistency andimprove deploymentefficiency.

● Reduce resource costs:Containers use systemresources moreefficiently. Comparedwith a VM with thesame configurations, acontainer can run moreapplications.

● Difficult to expandthe entirearchitecture of anapplication. As thecode size increases,code update andmaintenance wouldbe complicated.

● Difficult to launchnew functions,languages,frameworks, andtechnologies.




Mode 2: Separate thecomponents that arefrequently upgraded orhave high requirementson auto scaling from anapplication, and thencontainerize thesecomponents.

● Progressivetransformation:Reconstructing theentire architectureinvolves a heavyworkload. This modecontainerizes only apart of components,which is easy to acceptfor customers.

● Flexible scaling:Applicationcomponents that havehigh requirements onauto scaling arecontainerized. Whenthe application needsto be scaled, you onlyneed to scale thecontainers, which isflexible and reduces therequired systemresources.

● Faster rollout of newfeatures: Applicationcomponents that arefrequently upgradedare containerized. Insubsequent upgrades,only these containersneed to be upgraded.This shortens the timeto market (TTM) ofnew features.

Need to decouple someservices.




Mode 3: Transform anapplication tomicroservices and thencontainerize themicroservices one byone.

● Independent scaling:After an application issplit into microservices,you can independentlyincrease or decreasethe number ofinstances for eachmicroservice.

● Increased developmentspeed: Microservicesare decoupled fromone another. Codedevelopment of amicroservice does notaffect othermicroservices.

● Security assurancethrough isolation: Foran overall application,if a securityvulnerability exists,attackers can use thisvulnerability to obtainthe permission to allfunctions of theapplication. However,in a microservicearchitecture, if a serviceis attacked, attackerscan only obtain theaccess permission tothis service, but cannotintrude other services.

● Breakdown isolation: Ifone microservicebreaks down, othermicroservices can stillrun properly.

Need to transform theapplication tomicroservices, whichinvolves a large numberof changes.

Mode 1 is used as an example in this tutorial to illustrate how to containerize anenterprise resource planning (ERP) system.

12.2 Containerizing an Entire ApplicationThis tutorial describes how to containerize an ERP system by migrating it from aVM to CCE.



No recoding or re-architecting is required. You only need to pack the entireapplication into a container image and deploy the container image on CCE.

IntroductionIn this example, the enterprise management application is developed byenterprise A. This application is provided for third-party enterprises for use, andenterprise A is responsible for application maintenance.

When a third-party enterprise needs to use this application, a suit of Tomcatapplication and MongoDB database must be deployed for the third-partyenterprise. The MySQL database, used to store data of third-party enterprises, isprovided by enterprise A.

Figure 12-1 Application architecture

As shown in Figure 12-1, the application is a standard Tomcat application, and itsbackend interconnects with MongoDB and MySQL databases. For this type ofapplications, there is no need to split its architecture. The entire application ispacked as an image, and the mongoDB database is deployed in the same imageas the Tomcat application. In this way, the application can be deployed orupgraded through the image.

● Interconnecting with the MongoDB database for storing user files.● Interconnecting with the MySQL database for storing third-party enterprise

data. The MySQL database is an external cloud database.

BenefitsIn this example, the application was deployed on a VM. During applicationdeployment and upgrade, a series of problems is encountered, but applicationcontainerization has solved these problems.

By using containers, you can easily pack application code, configurations, anddependencies and convert them into easy-to-use building blocks. This achieves theenvironmental consistency and version management, as well as improves thedevelopment and operation efficiency. Containers ensure quick, reliable, andconsistent deployment of applications and prevent applications from beingaffected by deployment environment.



Table 12-2 Comparison between the tow deployment modes

Item Before: ApplicationDeployment on VM

After: Application DeploymentUsing Containers

Deployment

High deployment cost.A VM is required fordeploying a system for acustomer.

More than 50% cost reduced.Container services achieve multi-tenant isolation, which allows youto deploy systems for differententerprises on the same VM.

Upgrade Low upgrade efficiency.During version upgrades, youneed to log in to VMs one byone and manually configurethe upgrades, which isinefficient and error-prone.

Per-second level upgrade.Version upgrades can be completedwithin seconds by replacing theimage version. In addition, CCEprovides rolling updates, ensuringzero service downtime duringupgrades.

Operationandmaintenance (O&M)

High O&M cost.As the number ofapplications deployed forcustomer grows, the numberof VMs that need to bemaintained increasesaccordingly, which requires alarge sum of maintenancecost.

Automatic O&MEnterprises can focus on servicedevelopment without payingattention to VM maintenance.

12.3 Containerization ProcessThe following figure illustrates the process of containerizing an application.



Figure 12-2 Process of containerizing an application

12.4 Analyzing the ApplicationBefore containerizing an application, you need to analyze the runningenvironment and dependencies of the application, and get familiar with theapplication deployment mode. For details, see Table 12-3.

Table 12-3 Application environment

Item Description

Runtimeenvironment

OS OS that the application runs on, such as CentOS or Ubuntu.In this example, the application runs on CentOS 7.1.



Item Description

Runtimeenvironment

The Java application requires Java Development Kit (JDK),the Go language requires GoLang, the web applicationrequires Tomcat environment, and the corresponding versionnumber needs to be confirmed.In this example, the web application of the Tomcat type isused. This application requires the runtime environment ofTomcat 7.0, and Tomcat requires JDK 1.8.

Dependencypackage

Understand required dependency packages, such asOpenSSL and other system software, and their versionnumbers.In this example, no dependency package is required.

Deploymentmode

Peripheralconfigurations

MongoDB database: In this example, the MongoDBdatabase and Tomcat application are deployed on the sameserver. Therefore, their configurations can be fixed and thereis no need to extract their configurations.

External services with which the application needs tointerconnect, such as databases and file systems.These configurations need to be manually configured eachtime you deploy an application on a VM. However, throughcontainerized deployment, environment variables can beinjected into a container, facilitating deployment.In this example, the application needs to interconnect withthe MySQL database. You need to obtain the databaseconfiguration file. The server address, database name,database login username, and database login password areinjected through environment variables.url=jdbc:mysql://Server address/Database name #Database connection URLusername=**** #Username for logging in to the databasepassword=**** #Password for logging in to the database

Applicationconfigurations

You need to sort out the configuration parameters, such asconfigurations that need to be modified frequently andthose remain unchanged during the running of theapplication.In this example, no application configurations need to beextracted.NOTE

To avoid frequent image replacement, you are advised to classifyconfigurations of the application.● For the configurations (such as peripheral interconnection

information and log levels) that are frequently changed, you areadvised to configure them as environment variables.

● For the configurations that remain unchanged, directly writethem into images.



12.5 Preparing the Application RuntimeAfter application analysis, you have gained the understanding of the OS andruntime required for running the application. You need to make the followingpreparations:

● Installing Docker: During application containerization, you need to build acontainer image. To do so, you have to prepare a PC and install Docker on it.

● Obtaining the base image tag: Determine the base image based on the OSon which the application runs. In this example, the application runs onCentOS 7.1 and the base image can be obtained from an open source imagerepository.

● Obtaining the runtime: Obtain the runtime of the application and theMongoDB database with which the application interconnects.

Installing DockerDocker is compatible with almost all operating systems. Select a Docker versionthat best suits your needs.

NO TE

SWR uses Docker 1.11.2 or later to upload images.You are advised to install Docker and build images as user root. Obtain the password ofuser root of the host where Docker is to be installed in advance.

Step 1 Log in as user root to the device on which Docker is about to be installed.

Step 2 Run the following commands to quickly install Docker on the device runningLinux:

curl -fsSL get.docker.com -o get-docker.sh

sh get-docker.sh

Step 3 Run the following command to query the Docker version:

docker versionClient:Version: 17.12.0-ceAPI Version:1.35...

Version indicates the version number.

----End

Obtaining the Base Image TagDetermine the base image based on the OS on which the application runs. In thisexample, the application runs on CentOS 7.1 and the base image can be obtainedfrom an open source image repository.

NO TE

Search for the image tag based on the OS on which the application runs.



Step 1 Visit the Docker website.

Step 2 Search for CentOS. The image corresponding to CentOS 7.1 is centos7.1.1503. Youneed to use this image name when compiling the Dockerfile.

Figure 12-3 Obtaining the CentOS version

----End

Obtaining the Runtime

In this example, the web application of the Tomcat type is used. This applicationrequires the runtime of Tomcat 7.0, and Tomcat requires JDK 1.8. In addition, theapplication must interconnect with the MongoDB database in advance.

NO TE

Download the environment required by the application.

Step 1 Download Tomcat, JDK, and MongoDB installation packages of the specificversions.



1. Download JDK 1.8.Download the latest version from https://www.oracle.com/java/technologies/jdk8-downloads.html.Download an earlier version from https://www.oracle.com/java/technologies/javase-java-archive-javase8-downloads.html.

2. Download Tomcat 7.0 from https://tomcat.apache.org/download-70.cgi.3. Download MongoDB 3.2 from https://fastdl.mongodb.org/linux/mongodb-

linux-x86_64-rhel70-3.2.9.tgz.

Step 2 Log in as user root to the device running Docker.

Step 3 Run the following commands to create the directory where the application is to bestored: For example, set the directory to apptest.

mkdir apptest

cd apptest

Step 4 Use Xshell to save the downloaded dependency files to the apptest directory.

Step 5 Run the following commands to decompress the dependency files:

tar -zxf apache-tomcat-7.0.82.tar.gz

tar -zxf jdk-8u151-linux-x64.tar.gz

tar -zxf mongodb-linux-x86_64-rhel70-3.2.9.tgz

Step 6 Save the enterprise application (for example, apptest.war) in the webapps/apptest directory of the Tomcat runtime environment.

NO TE

apptest.war is used as an example only. Use your own application for actual configuration.

mkdir -p apache-tomcat-7.0.82/webapps/apptest

cp apptest.war apache-tomcat-7.0.82/webapps/apptest

cd apache-tomcat-7.0.82/webapps/apptest

./../../../jdk1.8.0_151/bin/jar -xf apptest.war

rm -rf apptest.war

----End

12.6 Compiling a Startup ScriptDuring application containerization, you need to prepare a startup script. Themethod of compiling this script is the same as that of compiling a shell script. Thestartup script is used to:

● Start up the software on which the application depends.● Set the configurations that need to be changed as the environment variables.



https://www.oracle.com/java/technologies/jdk8-downloads.html

https://www.oracle.com/java/technologies/jdk8-downloads.html

https://www.oracle.com/java/technologies/javase-java-archive-javase8-downloads.html

https://www.oracle.com/java/technologies/javase-java-archive-javase8-downloads.html

NO TE

Startup scripts vary according to applications. You need to compile the script based on yourservice requirements.

Procedure

Step 1 Log in as the root user to the device running Docker.

Step 2 Run the following commands to create the directory where the application is to bestored:

mkdir apptest

cd apptest

Step 3 Compile a script file. The name and content of the script file vary according toapplications. You need to compile the script file based on your application. Thefollowing example is only for your reference.

vi start_tomcat_and_mongo.sh#!/bin/bashsource /etc/profile./usr/local/mongodb/bin/mongod --dbpath=/usr/local/mongodb/data --logpath=/usr/local/mongodb/logs --port=27017 –forksed -i "s|mysql://.*/awcp_crmtile|mysql://$MYSQL_URL/$MYSQL_DB|g" /root/apache-tomcat-7.0.82/webapps/awcp/WEB-INF/classes/conf/jdbc.propertiessed -i "s|username=.*|username=$MYSQL_USER|g" /root/apache-tomcat-7.0.82/webapps/awcp/WEB-INF/classes/conf/jdbc.propertiessed -i "s|password=.*|password=$MYSQL_PASSWORD|g" /root/apache-tomcat-7.0.82/webapps/awcp/WEB-INF/classes/conf/jdbc.propertiesbash /root/apache-tomcat-7.0.82/bin/catalina.sh run

Script description:

● source /etc/profile: Load system environment variables.● ./usr/local/mongodb/bin/mongod --dbpath=/usr/local/mongodb/data --

logpath=/usr/local/mongodb/logs --port=27017 –fork: Start the MongoDBdatabase. In this example, the data storage directory is /usr/local/mongodb/data, and this directory will be used in the subsequent storage configuration.

● sed -i "s|...: These three script commands indicate that the contents related tothe MySQL database in the environment variables are written into theconfiguration file when Docker is started.

● bash /root/apache-tomcat-7.0.82/bin/catalina.sh run: Start Tomcat at the end.

----End

12.7 Compiling the DockerfileAn image is the basis of a container. A container runs based on the contentdefined in the image. An image has multiple layers. Each layer includes themodifications made based on the previous layer.

Generally, Dockerfiles are used to customize images. Dockerfile is a text file andcontains various instructions. Each instruction is used to build an image layer. Thatis, each instruction describes how to build an image layer.

This section describes how to compile a Dockerfile file.



NO TE

Dockerfiles vary according to applications. Dockerfiles need to be compiled based on actualservice requirements.

For details on how to write a quality Dockerfile, see Writing a Quality Dockerfile.

Procedure


Step 2 Write a Dockerfile.

vi Dockerfile

The following provides an example Dockerfile:

FROM centos:7.1.1503 #Indicates that centos:7.1.1503 is used as the base image.RUN mkdir -p /usr/local/mongodb/data \ #Create a folder for storing data and dependency files. You are advised to combine multiple commands into one to reduce the image size. && mkdir -p /usr/local/mongodb/bin \ && mkdir -p /root/apache-tomcat-7.0.82 \ && mkdir -p /root/jdk1.8.0_151COPY ./apache-tomcat-7.0.82/root/apache-tomcat-7.0.82 #Copy the files in the apache-tomcat-7.0.82 directory to the container directory.COPY ./jdk1.8.0_151 /root/jdk1.8.0_151 #Copy the files in the jdk1.8.0_151 directory to the container directory.COPY ./start_tomcat_and_mongo.sh /root/ #Copy start_tomcat_and_mongo.sh to the /root/ directory of the container. RUN chown root:root -R /root \ && echo "JAVA_HOME=/root/jdk1.8.0_151 " >> /etc/profile \ #Input Java environment variables. && echo "PATH=\$JAVA_HOME/bin:$PATH " >> /etc/profile \ && echo "CLASSPATH=.:\$JAVA_HOME/lib/dt.jar:\$JAVA_HOME/lib/tools.jar" >> /etc/profile \ && chmod +x /root \ && chmod +x /root/start_tomcat_and_mongo.sh ENTRYPOINT ["/root/start_tomcat_and_mongo.sh"] #When the container is started, commands in the start_tomcat_and_mongo.sh are automatically run. One or more commands can be run. A script can also be run.

In the preceding information:

● FROM statement: indicates that centos:7.1.1503 is used as the base image.

● Run statement: indicates that a shell command is executed in the container.

● Copy statement: indicates that files in the local computer are copied to thecontainer.

● ENTRYPOINT statement: indicates the commands that are run after thecontainer is started.

----End

12.8 Building and Uploading an ImageThis section describes how to build an entire application into a Docker image.After building an image, you can use the image to deploy and upgrade theapplication. This reduces manual configuration and improves efficiency.



https://support.huaweicloud.com/en-us/bestpractice-swr/swr_bestpractice_0002.html

NO TE

When building an image, ensure that files used to build the image are stored in the samedirectory.

Required Cloud Services

Software Repository for Container (SWR) provides easy, secure, and reliablemanagement over container images throughout their lifecycle, facilitating thedeployment of containerized services.

Basic Concepts● Image: A Docker image is a special file system that includes everything

needed to run containers: programs, libraries, resources, settings, and so on. Italso includes corresponding configuration parameters (such as anonymousvolumes, environment variables, and users) required within a containerruntime. An image does not contain any dynamic data, and its contentremains unchanged after being built.

● Container: A container is an entity that runs an image. The relationshipbetween an image and a container is similar to that between a class and aninstance in the object-oriented program design. A container can be created,started, stopped, deleted, or suspended.

Procedure


Step 2 Enter the apptest directory.

cd apptest

ll

Ensure that files used to build the image are stored in the same directory.

Step 3 Build an image.

docker build -t apptest .

Step 4 Upload the image to SWR. For details, see Uploading an Image Through aContainer Engine Client.

----End



https://www.huaweicloud.com/en-us/product/swr.html

https://support.huaweicloud.com/en-us/usermanual-swr/swr_01_0011.html


12.9 Creating a Container WorkloadThis section describes how to deploy a workload on CCE. When using CCE for thefirst time, create an initial cluster and add a node into the cluster.

NO TE

Containerized workloads are deployed in a similar way. The difference lies in:● Whether environment variables need to be set.● Whether cloud storage is used.

Required Cloud Services● Cloud Container Engine (CCE): a highly reliable and high-performance

service that allows enterprises to manage containerized applications. Withsupport for Kubernetes-native applications and tools, CCE makes it simple toset up an environment for running containers in the cloud.

● Elastic Cloud Server (ECS): a scalable and on-demand cloud server. It helpsyou to efficiently set up reliable, secure, and flexible application environments,ensuring stable service running and improving O&M efficiency.

● Virtual Private Cloud (VPC): an isolated and private virtual networkenvironment that users apply for on HUAWEI CLOUD. You can configure theIP address ranges, subnets, and security groups, as well as assign elastic IPaddresses and allocate bandwidth in a VPC.

Basic Concepts● A cluster is a collection of computing resources, including a group of node

resources. A container runs on a node. Before creating a containerizedapplication, you must have an available cluster.

● A node is a virtual or physical machine that provides computing resources.You must have sufficient node resources to ensure successful operations suchas creating applications.

● A workload indicates a group of container pods running on CCE. CCE supportsthird-party application hosting and provides the full lifecycle (fromdeployment to O&M) management for applications. This section describeshow to use a container image to create a workload.

Procedure

Step 1 Prepare the environment as described in Table 12-4.



https://www.huaweicloud.com/en-us/product/cce.html

https://www.huaweicloud.com/en-us/product/ecs.html

https://www.huaweicloud.com/en-us/product/vpc.html

Table 12-4 Preparing the environment

No. Item Procedure

1 Creating aVPC

Create a VPC before you create a cluster. A VPC providesan isolated, configurable, and manageable virtualnetwork environment for CCE clusters.If you have a VPC already, skip to the next task.1. Log in to the HUAWEI CLOUD console.2. In the service list, choose Network > Virtual Private

Cloud.3. On the Dashboard page, click Create VPC.4. Follow the instructions to create a VPC. Retain default

settings for parameters unless otherwise specified.

2 Creating akey pair

Create a key pair before you create a containerizedapplication. Key pairs are used for identity authenticationduring remote login to a node. If you have a key pairalready, skip this task.1. Log in to the HUAWEI CLOUD console.2. In the service list, choose Computing > Elastic Cloud

Server.3. In the navigation pane, choose Key Pair. On the Key

Pair page, click Create Key Pair.4. Enter a key pair name and click OK.5. In the displayed dialog box, click OK.

View and save the key pair. For security purposes, a keypair can be downloaded only once. Keep it secure toensure successful login.

Step 2 Create a cluster.1. Log in to the CCE console, choose Dashboard in the navigation pane, and

click Buy Cluster.2. Set Billing Mode to Pay-per-use, select an enterprise project, enter the

cluster name cluster-01, retain the default values for other parameters, andclick Next: Create Node.

3. Configure parameters for nodes in the cluster. Configure EIP and Login Modeas follows, and retain the default values for the other parameters.– EIP: Select Automatically assign to make the node accessible from

public networks. Retain the default values for other parameters.– Login Mode: Select Key pair, and select the created key pair for logging

in to the node.4. Click Next: Install Add-on and install the desired add-ons in the cluster. The

system resource add-ons are mandatory whereas the advanced functionaladd-ons are optional.

5. Click Next: Confirm. Read the product instructions and select I am aware ofthe above limitations. Confirm the configured parameters, specifications, andfees.




6. Click Submit. It takes about 6 to 10 minutes to create a cluster. View thecluster creation process as prompted.

7. Choose Resource Management > Nodes in the navigation pane. You canview the node whose status is Available and has bound with an elastic IPaddress.

Step 3 Deploy a workload on CCE.

1. In the navigation pane, choose Workloads > Deployments. On the pagedisplayed, click Create Deployment.

2. Set the following parameters, and retain the default settings for otherparameters:

– Workload Name: Set it to apptest.

– Cluster Name: Enter the name of the cluster created in Step 2.

– Instances: Set it to 1.

3. Click Next: Add Container. Click Add Container. On the My Images tabpage, select the image uploaded in Building and Uploading an Image andclick OK.

4. Retain the default settings for image parameters. Expand the EnvironmentVariables section, set the environment variables used for interconnecting withthe MySQL database. These environment variables are set in Compiling aStartup Script.

NO TE

In this example, interconnection with the MySQL database is implemented throughconfiguring the environment variables. Determine whether to use environmentvariables based on your service requirements.

Table 12-5 Configuring environment variables.

Variable Name Variable Value/Variable Reference

MYSQL_DB Database name.

MYSQL_URL IP address and port number of the database.

MYSQL_USER Database username.

MYSQL_PASSWORD

Database user password.

5. Expand the Data Storage section to configure cloud storage. To implementpersistent data storage, you need to configure cloud storage.

NO TE

In this example, the MongoDB database is used and persistent data storage is alsoneeded, so you need to configure cloud storage. Determine whether to use cloudstorage based on your service requirements.

In this example, set Container Path as the path to the MongoDB storagedirectory that you configured in the Docker startup script.



Figure 12-4 Configuring cloud storage

6. Retain the default values of other parameters and click Next: Set ApplicationAccess to configure workload access.

7. Click Add Service, set workload access parameters, and click OK.

NO TE

In this example, the application will be accessible from public networks by using anelastic IP address.

– Access Type: In this example, select NodePort.– Service Name: name of the application that can be accessed externally.

In this example, this parameter is set to apptest.– Service Affinity

▪ Cluster level: The IP addresses and access ports of all nodes in acluster can be used to access the workload associated with theService. Service access will cause performance loss due to routeredirection, and the source IP address of the client cannot beobtained.

▪ Node level: Only the IP address and access port of the node wherethe workload is located can be used to access the workloadassociated with the Service. Service access will not causeperformance loss due to route redirection, and the source IP addressof the client can be obtained.

– Port Settings

▪ Protocol: Set it to TCP.

▪ Container Port: port that the application will listen on the container.In this example, this parameter is set to 8080.

▪ Access Port: If this parameter is set to Automatically generated,the system automatically opens a real port on all nodes in thecurrent cluster and then maps the port number to the container port.



8. Click Next: Configure Advanced Settings. Skip the advanced settings, andclick Create.After the workload is created, you can view the running workload in theworkload list.

----End



13 Containerizing a Game Application(WOW)

13.1 OverviewThis document provides best practices for CCE to guide you through containerizinggame applications.

The features of the gaming industry, especially mobile games that are popularnowadays, are short, flat, and fast.

● Short: Games, especially mobile games, feature a short lifecycle, whichgenerally lasts for only one year.

● Flat: Game development relies on a flat background architecture. For mostgames, one server is deployed for one region. One machine and one databasecan serve multiple game servers.

● Fast: Game players generally increase in an explosive way. However, a gameserver supports only a limited number of users. A new server must bedeployed if the number of users reaches the upper limit. Due to a shortlifecycle, it is necessary that a game can be developed and rolled out in ashort time.

What Is a Container?

A container is a lightweight OS-layer virtualization technology. It allows userspace in an OS to be divided into several independent units running in the kernel,each of which is independent from each other. Such independent space is called acontainer.

With the development of virtualization technologies, one physical machine can bevirtualized into multiple VMs, but virtualizing an independent OS incurs systemloss, limiting the number of target VMs. In contrast, one machine can run dozensor even hundreds of containers with basically no performance loss. Inaddition, starting a container is as simple as starting a process, which can becompleted in seconds. In gaming scenarios, the container technology is extremelyadvantageous.

Cloud Container EngineBest Practice 13 Containerizing a Game Application (WOW)


Why Is a Container Preferred?● A traditional game background architecture has the following

disadvantages: One machine runs a large number of game servers at thesame time. When the machine breaks down, users are widely affected.

Container solution: One of the most important advantages of the containertechnology lies in its lightweight, which supports virtualization of anindependent system operating environment with a fine granularity. Thisenables a physical server, or a cloud server, to run hundreds of thousands ofindependent containers at the same time. Each kind of service logic in agame, such as marching, fighting, and chatting control logic, can run inan independent container. This series of containers constitutes anindependent game world.

In addition, the resource usage of these containers can be properly plannedbased on service types, so that the resources are isolated for differentcontainers.

Moreover, after containerization, a machine failure may affect only someservice logic in some game servers. For example, when the machine runningthe container that controls marching tasks is faulty, frame freezing occursinstantaneously but the marching tasks recover immediately. Usingmonitoring methods, some important service logic can be run on multiplebackup containers at the same time, and the service logic can be quicklyswitched to the backup containers in case of unavailability.

● Games feature a short lifecycle and require quick development androllout.

Container solution: The core concepts of Docker containers are Build, Ship,and Run, covering the entire process from development to deployment. Afterthe development is complete, containers are packaged into container imagesand stored in a repository before testing. After the testing is complete, thecontainer images are stored in the repository again and finally deployed inthe production environment. The three phases are smoothly connected,avoiding the workload of setting up a complex running environment. In thisway, games can be quickly developed and rolled out.

● The number of game players dramatically fluctuates, requiring autoscaling of game applications in a short time.

Container solution: A container can be started in seconds. When the numberof game players dramatically increases, containers can be started in secondsto ensure service stability.

Core Advantages of CCE● Advanced bare-metal container service, improving the gaming service

performance by more than 200%

The gaming industry places stringent requirements on network performanceand computing capabilities of servers. CCE supports a bare-metal containerservice, as shown in Figure 13-1. Games can be deployed based on BareMetal Server (BMS), and containers can directly run on physical machines. Byusing containers, excellent performance can be achieved without anyperformance loss caused by virtualization, thereby improving the gamingservice performance by more than 200%.



Figure 13-1 BMS container service

● Auto scaling of containers in seconds, saving a lot of resource costs

Traffic unpredictability has become a "New Normal" of game applications.CCE supports auto scaling in seconds to ensure high service stability andimprove user experience. In addition, reserved resources are reduced, savinginvestments by millions of dollars. Based on features of games, CCE providesflexible auto scaling policies, which can be selected and combined for use asrequired.

Table 13-1 Flexible auto scaling policies

Recommended AutoScaling Policy

Description

● The number ofgame playersdramaticallyfluctuates every day,so the PeriodicPolicy isrecommended.

For example, the number of players of a gamereaches the peak in the afternoon and evening ona given day.You are advised to use the Periodic Policy. Forexample, 100 pods are added at 13:00 every dayfrom January 1, 2018 to January 1, 2019.

● For new games, theMetric-based Policyis recommended.

After a new game is released, it is uncertainwhether how many players will play this game. Itis difficult to reserve a proper number ofmachines based on existing experience.You are advised to use the Metric-based Policy.For example, if the CPU or memory usage exceeds70%, one pod is added. If the CPU or memoryusage is less than 40%, one pod is reduced.

● For various in-gameevents, theScheduled Policy isrecommended.

In-game events are frequently held. You areadvised to configure the Scheduled Policy beforean event begins. For example, 100 pods are addedat 12:00 on August 8.

● Rolling upgrade policy, causing no service interruption and ensuring lag-free experience of game players



Game requirements grow rapidly and versions change frequently. The upgradeefficiency and user experience during the upgrade are crucial for gamingservices.CCE provides a rolling upgrade policy to separately update pods one by oneinstead of updating all pods at the same time. This ensures that services arenot interrupted during the upgrade.

● Support for stateful applications, solving the problem of containerizinggame applicationsGame servers constitute an independent game world. In this world, data ofplayers needs to be continuously updated and stored. The prerequisite forcontainerizing game applications is to ensure data storage. CCE supportsstateful containerized applications (applications that store data or statusesduring the running), and leverages high-availability volumes by means ofstorage capabilities such as Elastic Volume Service (EVS) and Scalable FileService (SFS) of HUAWEI CLOUD. This solves the problem of containerizinggame applications.

13.2 Deployment ProcessThis tutorial uses a game as an example to describe how to deploy a gameapplication on CCE and demonstrate how to scale and upgrade this application.



Figure 13-2 Deployment process

13.3 Preparation: Reconstructing the Game ApplicationArchitecture

Three application containerization modes are available. For details, click here. Thissection describes reconstruction on the architecture of a game application intomicroservices before the game application is containerized.

Only overall reconstruction suggestions are provided in this section, and thereconstruction process is not described in detail. For details about thereconstruction process, click here.



https://www.huaweicloud.com/en-us/consultation/

NO TE

This section provides suggestions on reconstruction of the game application architecture.You do not need to perform any operation. To perform operations, go to Analyzing theGame Application.

Suggestions on Containerization ReconstructionThe original game application architecture is as follows:

Figure 13-3 Original architecture

As shown in Figure 13-3,

● The game application consists of three components: wow-auth loginauthentication system, wow-world game server, and MySQL database.

● The wow-auth login authentication system and its dependency are installedon one VM, the MySQL database and its dependency are installed on two ormore VMs, and the wow-world game server and its dependency are installedon three or more VMs. In the current architecture, if there are multiple gameservers, they must be installed on multiple VMs. In this case, multiple VMsmust be prepared, and independence packages required by differentcomponents must be installed on each VM, featuring heavy workload.

● This architecture causes poor scalability and difficult scaling, and brings highcosts in maintenance. A new VM must be installed before a game server isadded. In addition, it is a difficult job to maintain multiple VMs.

● Difficult upgrade: Upgrading the VMs require you to upgrade theconfigurations of these VMs one by one, which is time-consuming and error-prone.

You are advised to reconstruct the original architecture as follows:



Figure 13-4 New architecture

As shown in Figure 13-4, the three components (including the wow-auth loginauthentication system, wow-world game server, and MySQL database) of thegame application are respectively containerized and deployed on VMs. The newfeature has the following advantages:

NO TE

In this tutorial, the MySQL database is used. During actual commercial use, select a properdatabase based on your service requirements.

● Easy deployment: The three components of the game application are madeinto container images, and the images are uploaded to a container imagerepository. You can directly deploy the containerized game application basedon these images by using CCE.

● Good scalability and quick scaling: To add a game server, you only need tostart a container. You can start a container in seconds.

● Easy upgrade: You can quickly upgrade the components by merely replacingthe images. In addition, CCE provides rolling upgrade, causing no serviceinterruption during the upgrade.

13.4 Analyzing the Game ApplicationBefore deploying the game application, learn the application components to bedeployed and the relationships between these components.



Figure 13-5 Game application architecture

The entire game application consists of four components: game client on theforeground, wow-auth login authentication system, wow-world game server, andMySQL database on the background. The following table describes thesecomponents and their relationships.

Table 13-2 Application environment

Item Description

Application Components

● Foreground: gameclient

The game client is prepared in advance. You candirectly download the client to install the game.

● Background: database(MySQL)

Stores game data.

● Background:authentication system(wow-auth)

Authenticates login of game players.

● Background: gameserver (wow-world)

Provides the game.

Relationships between the components

● Both wow-auth and wow-world need to connect to the MySQL database fordata storage. In this example, they are connected by using environmentvariables.

● wow-auth needs to connect to wow-world. In this example, they areconnected through environment variables.

13.5 Preparing the EnvironmentBefore deploying the game application, prepare the hardware and HUAWEICLOUD environment, including:



● Hardware: Prepare a Windows PC with a graphics card and at least 20 GBdisk space for running the game client.

NO TE

You do not need to prepare a Windows PC if you only want to learn how to deploy thegame but do not run the game.

● Game application images: This game application consists of threecomponents: wow-auth login authentication system, wow-world game server,and MySQL database. CCE supports deployment of the MySQL database withjust a few clicks, so you do not need to prepare the image of the MySQLdatabase. Instead, prepare the images of the wow-auth login authenticationsystem and wow-world game server.

NO TE

In this tutorial, the images of the two components have been created, so you candirectly download them. For details on how to create an image, see the relateddescriptions in Containerizing an Enterprise Application (ERP).

Hardware EnvironmentPrepare a Windows PC with a graphics card and at least 20 GB disk space forrunning the game client.

Game Application ImagesAs shown in Figure 13-4, this game application consists of three components:wow-auth login authentication system, wow-world game server, and MySQLdatabase.

CCE supports deployment of the MySQL database with just a few clicks, so you donot need to prepare the image of the MySQL database. Instead, prepare theimages of the wow-auth login authentication system and wow-world game server.

NO TE

In this tutorial, the images of the two components have been created, so you can directlydownload them. For details on how to create an image, see the related descriptions inContainerizing an Enterprise Application (ERP).

Step 1 Buy a HUAWEI CLOUD ECS for downloading and uploading images.

1. Log in to the management console, and set the region to CN North-Beijing1in the upper left corner.

2. In the service list, choose Computing > Elastic Cloud Server, and click BuyECS in the upper right corner.

3. Set the parameters listed in Table 13-3 on the displayed Buy ECS page. Forthe other parameters, retain their default values.



Table 13-3 Buying ECS


Billing Mode To save costs, you are advised to select Pay-per-use. Youcan clear the resources after deploying the gameapplication.

Image For Public image, you are advised to select Ubuntu 16.04server 64bit (40 GB), which is the same as the OS used forcompiling games.

Login Mode Set a password with high security level.

ECS Name You can use the ECS name that is automatically generated.In this example, the ECS name is changed to ecs-test.

4. After the configuration, click Buy Now. On the page displayed, confirm your

order and click Submit.5. After the ECS is created, you can view it in the ECS list and its status is

Running.

6. Click next to the EIP in the IP Address column to obtain the EIP.

Figure 13-6 Obtaining the elastic IP address

Step 2 Use a remote login tool, such as Xshell, to log in to the ECS.

ssh root@elastic IP address bound to the ECS

Step 3 Run the following command to install Docker:

curl -fsSL get.docker.com -o get-docker.sh

sh get-docker.sh

It takes about five minutes to install Docker.

Step 4 Log in to the Software Repository for Container (SWR) console. In the navigationpane, choose My Image and click Upload Through Docker Client. In step 2, clickGenerate a temporary Docker login command. Then, copy this command andrun it on the node with Docker installed.



Figure 13-7 Generating a temporary Docker login command

The command is successfully executed if the following information is displayed:Login Succeeded

Step 5 Pull the prepared four images, including two images for the wow-authauthentication system (versions 5.0 and 5.1) and another two images for thewow-world game server (versions 5.0 and 5.1). Two versions of images areprepared to demonstrate subsequent upgrade operations. Run the followingcommands to pull images:

NO TE

It takes about 5 to 10 minutes to pull the images.

docker pull swr.cn-north-1.myhuaweicloud.com/wow/wow:wowauth-5.0

docker pull swr.cn-north-1.myhuaweicloud.com/wow/wow:wowworld-5.0-withmap

docker pull swr.cn-north-1.myhuaweicloud.com/wow/wow:wowauth-5.1

docker pull swr.cn-north-1.myhuaweicloud.com/wow/wow:wowworld-5.1-withmap

Step 6 Run the following command to view the images:

docker images

Step 7 Run the following command to upload the wowauth-5.0 image to SWR:

docker tag [Image name:Tag] swr.cn-north-4.myhuaweicloud.com/[Organizationname]/[Image name:Tag]

Example:

NO TE

● In the example command, gametest indicates the organization name, which must beglobally unique. If an organization has been created on SWR, you are advised to use thename of the existing organization.

● cn-north-4 in swr.cn-north-4.myhuaweicloud.com indicates the region where the ECSfor uploading images is located.

docker tag swr.cn-north-1.myhuaweicloud.com/wow/wow:wowauth-5.0swr.cn-north-4.huaweicloud.com/gametest/wow:wowauth-5.0



docker push swr.cn-north-4.myhuaweicloud.com/gametest/wow:wowauth-5.0

Upload the wowauth-5.1, wowworld-5.0-withmap, and wowworld-5.1-withmap images in the same way.

Step 8 On the SWR console, choose My Images in the navigation pane, and click wow inthe image list. Then, you can view the four image versions on the page that isdisplayed.

Figure 13-8 My Images

----End

13.6 Deploying the Game ApplicationTo deploy the game application on CCE, you need to perform the followingoperations:

1. Create a cluster: A container cluster is a logical group that runs applicationsand contains a group of cloud server resources. Each cluster node correspondsto a cloud server. When using CCE for the first time, create an initial clusterand add a node into the cluster.

2. Deploy the MySQL database: Deploy the distributed MySQL service by usingcharts provided by CCE with a few clicks.

3. Deploy the wow-auth authentication system: Deploy the wow-authauthentication system on CCE.



4. Deploy the wow-world game server: Deploy the wow-world game server onCCE.

Creating a ClusterA container cluster is a logical group that runs applications and contains a groupof cloud server resources. Each cluster node corresponds to a cloud server. Whenusing CCE for the first time, create an initial cluster and add a node into thecluster.

Step 1 Before creating a cluster, create a VPC and a key pair.

NO TE

If a VPC and a key pair are available, skip to the next step.

Table 13-4 Preparing the environment

No. Item Procedure

1 Creating aVPC

Create a VPC before you create a cluster. A VPC providesan isolated, configurable, and manageable virtualnetwork environment for CCE clusters.If you have a VPC already, skip to the next task.1. Log in to the HUAWEI CLOUD console.2. In the service list, choose Network > Virtual Private

Cloud.3. On the Dashboard page, click Create VPC.4. Follow the instructions to create a VPC. Retain default

settings for parameters unless otherwise specified.

2 Creating akey pair

Create a key pair before you create a containerizedapplication. Key pairs are used for identity authenticationduring remote login to a node. If you have a key pairalready, skip this task.1. Log in to the HUAWEI CLOUD console.2. In the service list, choose Computing > Elastic Cloud

Server.3. In the navigation pane, choose Key Pair. On the Key

Pair page, click Create Key Pair.4. Enter a key pair name and click OK.5. In the displayed dialog box, click OK.

View and save the key pair. For security purposes, a keypair can be downloaded only once. Keep it secure toensure successful login.

Step 2 Log in to the CCE console, choose Dashboard in the navigation pane, and clickBuy Cluster.

Step 3 Set Cluster Name to cluster-wow, retain the default settings for otherparameters, and click Next.




Step 4 Set the parameters for adding a node into the cluster. Set the node specifications,network, and login parameters as follows, and retain the default settings for theother parameters:● Specifications: 4 vCPUs and 8 GB memory.

NO TE

The 4-core CPU and 8 GB memory are the minimum specifications for the gameapplication, supporting deployment of only one game server. To deploy more gameservers, scale out the node or configure higher specifications.

● EIP: Select Automatically assign to make the node accessible from publicnetworks. Retain the default values for other parameters.

● Login Mode: Select Key Pair, and select the key pair you created in Table13-4 for logging in to the node.

Step 5 Click Buy Now. Review the order details, and click Submit.

It takes about 6 to 10 minutes to create a cluster. View the cluster creation processas prompted.

Step 6 Choose Resource Management > Nodes in the navigation pane. You can view thenode whose status is Available and has bound with an elastic IP address.

----End

Deploying the MySQL Database

Deploy the distributed MySQL service by using charts provided by CCE with a fewclicks. For this game application, you only need to apply for one MySQL databasefor the wow-auth authentication system and wow-world game server. In actualapplication scenarios, deploy the MySQL service based on your servicerequirements.

Step 1 Log in to the CCE console.

Step 2 Choose Charts > Sample Charts in the navigation pane, and click Install Chartunder mysql-ndb.

NO TE

Click mysql-ndb to view the introduction and architecture of the chart.

Figure 13-9 Installing mysql-ndb

Step 3 Set the basic information of the database.● Release Name: Specify the workload name, for example, wow-mysql.● Chart Name: mysql-ndb, which cannot be modified.




● Chart Version: 1.0.0.● Cluster: Select the cluster created in Creating a Cluster.● Namespace: Select a namespace based on service requirements.● Workload Deployment Specifications: 1X.● Database Name: clustertest.● Normal User Name of Database: testuser. The name is pre-set in the image.● Normal User Login Password of Database: This parameter cannot be left

blank. Set a password and keep it secure.● Administrator Password: This parameter cannot be left blank. Set a

password and keep it secure.

Step 4 Click Install at One Click. On the displayed Confirm page, confirm your order andclick Submit.

Click to return to the template instance list. In the template instance list, you cansee that the wow-mysql application is installed successfully.

Step 5 Click wow-mysql. You can view the three instances of MySQL. If their statusesturn to Running, the wow-mysql application has been successfully created. Thecreation takes about 5 minutes.● wow-mysql-mgmd: management workload of MySQL, which is used to

manage other MySQL components.● wow-mysql-ndbd: stores data.● wow-mysql-mysqld: traditional MySQL server that uses the NDB Cluster

storage engine. The instance accesses MySQL application data from the dataapplication layer.

Step 6 Click wow-mysql-mysqld to access its details page. Click the Services tab, andobtain and record the Access Address.

Figure 13-10 Obtaining the access address

----End

Deploying the wow-auth Authentication SystemDeploy the wow-auth authentication system on CCE. During the deployment, setthe wow-auth authentication system to connect to the MySQL database andwow-world game server through environment variables.




Step 2 In the navigation pane, choose Workloads > Deployments, and click CreateDeployment.

NO TE

For the game application described in this tutorial, both its authentication system andgame server are Deployments. You are advised to reconstruct them into Deployments ifthey are StatefulSets in actual application scenarios. The differences between Deploymentsand StatefulSets are as follows:● Deployment: a workload that does not store any data or statuses, for example, Nginx.● StatefulSet: a workload that stores data or statuses during running. For example,

MySQL is a StatefulSet because it needs to store new data.

Step 3 Set the basic information about the workload.● Workload Name: Specify the workload name, for example, wow-auth.● Cluster Name: Select the cluster created in Creating a Cluster.● Namespace: Retain the default value.● Instances: You are advised to set the quantity to 1. Otherwise, resources may

be insufficient.● Description: You can leave it blank.

Step 4 Click Next, and click Add Container. On the Select Container Image pagedisplayed, search for wow, and select the wowauth-5.0 image version.

Step 5 Retain the default values for the other parameters, and set the parameters forContainer Resources as required. After the setting, select the 0.5X specifications.

NO TE

Configure compute resources based on the application requirements. In this tutorial, thewow-auth component of the game application requires at least 0.5 CPU core and 0.5 GiBmemory.

Figure 13-11 Configuring container resources




Step 6 Expand Environment Variables and configure the environment variables forconnecting the wow-auth authentication system to the MySQL database andwow-world game server.

NO TE

The environment variables listed in the following list are pre-set in the image. If you areunclear about the settings, see How Can I Obtain the Values of Environment VariablesWhen Deploying a Game Application?.

Table 13-5 Setting environment variables

VariableName

Description Variable Value/VariableReference

mysqlip Set this variable to the databaseaccess address obtained in Step6.

10.247.59.224//10.247.130.188

mysqlrootpasswd

Password of the databaseadministrator, which must bethe same as the administratorpassword set in Step 3.

-

biboaddress

External access address of thegame server. You can use theelastic IP address or elastic loadbalancer mode. Elastic IPaddress mode is used in thistutorial. Select the elastic IPaddress queried in Step 6.

10.3.2.119

biboport External access port of thegame server. You must set avalue ranging from 30000 to32767 in advance.NOTE

This value must be globally uniquein the current cluster. In thistutorial, for easy operations, a fixedvalue is specified for connection tothe game server. During actualapplication deployment, you areadvised to specify the dependencieswhen creating the images.

32500

Step 7 Click Next: Set Application Access, and click Add Service to configure a policy bywhich the workload will be accessed externally.



Figure 13-12 Adding a service

● Access Type: In this example, select NodePort.

● Service Name: The service name can be the same as the application name,for example, wow-auth.

● Service Affinity: In this example, the cluster level is selected.

– Cluster level: The IP addresses and access ports of all nodes in a clustercan be used to access the workload associated with the Service. Serviceaccess will cause performance loss due to route redirection, and thesource IP address of the client cannot be obtained.

– Node level: Only the IP address and access port of the node where theworkload is located can be used to access the workload associated withthe Service. Service access will not cause performance loss due to routeredirection, and the source IP address of the client can be obtained.

● Port Settings

– Protocol: In this example, TCP is selected.

– Container Port: Listening port of the authentication system, which is port3724 in this example. Do not change the port number because the portnumber 3724 has been set in the image.

– Access Port: Node port (with a private IP address) to which the containerport will be mapped. In this example, the port is automatically generated.

Step 8 Click OK.

Step 9 Click Next: Configure Advanced Settings. Skip the advanced settings, and clickCreate.

On the workload list page, you can view the created workload. It takes about oneminute to create the workload.

----End



Deploying the wow-world Game ServerDeploy the wow-world game server on CCE. During the deployment, set the wow-world game server to connect to the MySQL database through environmentvariables.


Step 2 In the navigation pane, choose Workloads > Deployments, and click CreateDeployment.

Step 3 Set the basic information about the workload.● Workload Name: Specify the workload name, for example, wow-world.● Cluster Name: Select the cluster created in Creating a Cluster.● Namespace: Retain the default value.● Instances: You are advised to set the quantity to 1. Otherwise, resources may

be insufficient.● Description: You can leave it blank.

Step 4 Click Next: Add Container, and click Add Container. On the Select ContainerImage page displayed, search for wow, and select the wowworld-5.0-withmapimage version.

Step 5 Configure Container Resources. Configure compute resources based on theapplication requirements. In this tutorial, the wow-world component of the gameapplication requires at least 2 CPU cores and 2 GiB memory.

Figure 13-13 Configuring container resources

Step 6 Set the environment variables used for interconnection with the MySQL database.Table 13-6 describes the variables.

NO TE

If you are unclear about the settings, see How Can I Obtain the Values of EnvironmentVariables When Deploying a Game Application?.

Table 13-6 Setting environment variables

VariableName


mysqlip Set this variable to the databaseaccess address.

10.247.59.224




VariableName


mysqlrootpasswd

Password of the databaseadministrator, which must bethe same as the administratorpassword set in Step 3.

-

Step 7 Click Next: Set Application Access, and click Add Service to configure the policyby which the workload will be accessed externally.

● Access Type: In this example, NodePort is selected.

● Service Name: The service name can be the same name as the workloadname, for example, wow-world.

● Service Affinity: In this example, the cluster level is selected.

– Cluster level: The IP addresses and access ports of all nodes in a clustercan be used to access the workload associated with the Service. Serviceaccess will cause performance loss due to route redirection, and thesource IP address of the client cannot be obtained.

– Node level: Only the IP address and access port of the node where theworkload is located can be used to access the workload associated withthe Service. Service access will not cause performance loss due to routeredirection, and the source IP address of the client can be obtained.

● Protocol: In this example, TCP is selected.

● Container Port: Listening port of the game server, which is port 8085 in thistutorial. Do not change the port number because the port number has beenset in the image.

● Access Port. This port must be the same as the biboport in the environmentvariable settings added when the authentication system is deployed. If nosuch environment variable is found, see How Can I Obtain the Access PortWhen Deploying a Game Application?.

Step 8 Click OK.

Step 9 Click Next. Skip the advanced settings, and click Create.

On the workload list page, you can view the created workload. It takes about oneminute to create the workload.

----End

13.7 Running the GameRun the deployed game application and log in to the game client.

Prerequisites

A Windows PC with a graphics card and at least 20 GB disk space is available.



Procedure

Step 1 Log in to the Windows PC.

Step 2 Download the wow game client. In this example, the matching version number iswow_cn_3.3.5.13930. This third-party download link is for reference only.

Step 3 Obtain the server address and the service port of the wow-auth authenticationsystem.● Server address: Log in to the CCE console, choose Resource Management >

Nodes in the navigation pane, and view the elastic IP address.

Figure 13-14 Obtaining the elastic IP address

● Service port of the wow-auth authentication system: Choose Workloads >Deployments in the navigation pane, click wow-auth to access its detailspage, and then click the Services tab page to obtain the service port.

Step 4 Configure the game client.

Open the Config.wtg file on the game client, and configure the following serverinformation in the file:

SET realmlist server address:service port of the wow-auth authentication system

Example:

SET realmlist 10.4.10.211:31739

Step 5 Download the login tool used to access the private server of World of WarcraftTeahouse, merge the tool into the wlkwowc directory, and double-click the clientprogram wow to enter the game.

Step 6 (When deploying your real game application) Connect to the registration API andlog in to the game application using your account to check whether the gameapplication functions properly.

----End

13.8 Scaling the Game ApplicationScale the wow-auth workload in manual or automatic mode by using CCE.

Manual Scaling

Step 1 Log in to the CCE console, choose Workloads > Deployments in the navigationpane, and click wow-auth to access its details page.



http://obs.cn-north-1.myhuaweicloud.com/cce.cn-north-1/master-images/CCE0.0.0.B001/wlkwowc.rar



Step 2 On the Scaling tab page, click under Manual Scaling, change Pods to 2, andclick Save.

Figure 13-15 Manual scaling

Step 3 On the Pods tab page of the details page, you can view that the new pod is beingcreated. The creation will be completed within seconds. If the pod status is not

refreshed, click . The status of the instance is Running.

----End

Auto Scaling


Step 2 On the Scaling tab page, click Add Scaling Policy under Auto Scaling.

CCE supports auto scaling in seconds to ensure high service stability and improveuser experience. In addition, reserved resources are reduced, saving investments bymillions of dollars. Based on features of games, CCE provides flexible auto scalingpolicies, which can be selected and combined for use as required.

Table 13-7 Flexible auto scaling policies


Description

Periodic Policy For example, the number of players of a gamereaches the peak in the afternoon and evening on agiven day.You are advised to use the Periodic Policy. Forexample, 100 pods are added at 13:00 every dayfrom January 1, 2018 to January 1, 2019.





Description

Metric-based Policy After a new game is released, it is uncertain whetherhow many players will play this game. It is difficultto reserve a proper number of machines based onexisting experience.You are advised to use the Metric-based Policy. Forexample, if the CPU or memory usage exceeds 70%,one instance is added. If the CPU or memory usageis less than 40%, one pod is reduced.

Scheduled Policy In-game events are frequently held. You are advisedto configure the Scheduled Policy before an eventbegins. For example, 100 pods are added at 12:00 onAugust 8.

Step 3 Add a metric-based policy. In this policy, one pod is added when the CPU usage isgreater than 70%.

Figure 13-16 Adding a metric-based policy

● Policy Name: Enter a policy name, for example, policy-01.

● Metric: Select CPU Usage.

● Policy Type: Select Metric-based Policy.

● Trigger Condition: The average CPU usage is greater than 70%.

● Monitoring window: Set a period for collecting metric statistics. Select avalue from the drop-down list. If the parameter is set to 60s, metric statisticsare collected every 60s.

● Threshold Crossings: If the parameter is set to 1, the configured action willbe triggered when the threshold is reached for one consecutive statisticalperiod.

● Action: Add one pod.



Step 4 Click OK. This policy is displayed under Auto Scaling. If the average CPU usage isgreater than 70%, this policy is triggered.

----End

13.9 Upgrading the Game ApplicationGame requirements grow rapidly and versions change frequently. The upgradeefficiency and user experience during the upgrade are crucial for gaming services.

CCE provides a rolling upgrade policy to separately update pods one by oneinstead of updating all pods at the same time. This ensures that services are notinterrupted during the upgrade.

This section uses wow-auth as an example to demonstrate the rolling upgrade ofapplications.

PrerequisitesEnsure that the workload to be upgraded has at least two pods. You are advisedto manually scale the workload into two pods before performing the upgrade.

Procedure


Step 2 Choose Workloads > Deployments in the navigation pane, and click wow-authto access its details page.

Step 3 Click the Upgrade tab. When the workload is being created, the rolling upgradewill be used by default.

Step 4 After containerization, you can easily upgrade the workload by replacing its image.Click Replace Image, select wowauth-5.1, and click OK.

Figure 13-17 Replacing the image

Step 5 Click Submit in the lower right corner of the page. In the dialog box displayed,click OK.

On the Pods tab page, you can view that one pod is being created and then theother is being stopped. This ensures that there is always a pod running and theservice is not interrupted during the upgrade.




Figure 13-18 Rolling upgrade

Step 6 Click on the right. If both pods are in the running state, the upgrade issuccessful.

----End

13.10 Deleting ResourcesYou have finished deploying the game application. Fees are generated during nodeand application running. When you finish deploying the game application, you areadvised to delete the resources associated with it to avoid incurring charges forresources that you are not using.

Procedure

Step 1 Delete the application.

1. Log in to the CCE console.2. In the navigation pane, choose Workloads > Deployments.3. On the page displayed, click More > Delete in the Operation column of

wow-auth and wow-world. Delete the workloads as prompted.4. Choose Workloads > StatefulSets. On the page displayed, click More >

Delete in the Operation column of wow-mysql-mgmd, wow-mysql-ndbd,and wow-mysql-mysqld. Delete the workloads as prompted.

Step 2 Delete cluster resources.

1. Log in to the CCE console.2. In the navigation pane, choose Resource Management > Clusters.3. Click More > Delete in the target cluster card view, and delete the cluster as

prompted.

Step 3 Delete the ECS on which Docker is installed.

1. Log in to the ECS console.2. Choose More > Delete in the Operation column of the newly created ecs-cy

server. Then, on the page displayed, select Release the EIPs bound to thefollowing ECSs and Delete the data disks attached to the following ECSs,and click OK to delete the ECS.

----End





https://console.huaweicloud.com/ecm/?locale=en-us#/ecs/manager/vmList

13.11 FAQs

How Can I Obtain the Values of Environment Variables When Deploying aGame Application?


Step 2 In the navigation pane, choose Workloads > StatefulSets.

Step 3 Click wow-mysql-mysqld to access its details page.

Step 4 Click the Services tab. The value of the mysqlip environment variable is obtainedfrom Figure 13-19.

Figure 13-19 Database IP address

Step 5 Click the Upgrade tab. The value of the mysqlrootpasswd environment variable isequal to the value of MYSQL_ROOT_PASSWORD in Figure 13-20.

Figure 13-20 Database administrator account password

Step 6 In the navigation pane, choose Resource Management > Nodes to view theelastic IP address of the node created in Creating a Cluster. The value ofbiboaddress is the elastic IP address.




Figure 13-21 Obtaining the EIP

----End

How Can I Obtain the Access Port When Deploying a Game Application?


Step 2 Click the Upgrade tab, expand Advanced Settings, and view the value ofbiboport.

Figure 13-22 Obtaining the access port

----End




14 Installing, Deploying, andInterconnecting Jenkins with SWR and CCE

Clusters

14.1 Overview

What Is Jenkins?Jenkins is an open source continuous integration (CI) tool that provides user-friendly GUIs. It originates from Hudson and is used to automate all sorts of tasksrelated to building, testing, and delivering or deploying software.

Jenkins is written in Java and can run in popular servlet containers such asTomcat, or run independently. It is usually used together with the version controltools (or SCM tools) and build tools.

Common version control tools include SVN and Git, and build tools include Maven,Ant, and Gradle.

Deployment Analysis● Jenkins is deployed in CCE clusters using the container image.● Jenkins pipelines can be interconnected with SWR by calling the docker build/

login/push commands in the pipeline. In addition, the long-term valid dockerlogin command needs to be obtained through SWR, which has beensupported.

● Jenkins pipelines can be interconnected with CCE by using Kubernetes add-ons. You can import multiple kubeconfig configuration files to connect todifferent users and clusters with different permissions in pipelines. Differentclusters have different kubeconfig files. Users with different permissions(RBAC) in Kubernetes can be associated with cluster roles through theirservice account, which generates dedicated kubeconfig files.

● After interconnecting with Kubernetes clusters, you can use YAML files todeploy and upgrade Kubernetes resources (such as Deployments, Services,ingresses, and jobs).


14 Installing, Deploying, and Interconnecting Jenkinswith SWR and CCE Clusters


14.2 Installing and Deploying Jenkins

Selecting an Image

Select a functionally stable image on Docker Hub.

In this test, the image jenkinszh/jenkins-k8s:2.239 is used.

Preparations● Before creating a containerized workload, you need to buy a cluster (the

cluster must contain at least one node with four vCPUs and 8 GB memory).For details, see Buying a Hybrid Cluster.

● To enable access to a workload from a public network, ensure that an elasticIP address (EIP) has been bound to or a load balancer has been configuredfor at least one node in the cluster.

Installing and Deploying Jenkins on CCE

Step 1 Log in to the CCE console, choose Workloads > Deployments in the navigationpane, and click Create Deployment on the right.

Step 2 On the Specify Basic Info page, set the workload name to jenkins (which can beset as required), set the number of pods to 1, and click Next: Add Container.

Step 3 Click Add Container.

Step 4 In the displayed Select Container Image dialog box, click Third-Party Images, setImage Address to jenkinszh/jenkins-k8s:2.239, retain the default values forother parameters, and click OK.

Figure 14-1 Selecting a third-party image

Step 5 Set Container Resources. In this example, set CPU Limit to 2 cores and MemoryLimit to 2048 MiB.




https://hub.docker.com/r/jenkinszh/jenkins-k8s on Docker Hub


Figure 14-2 Setting container specifications

Step 6 Click Data Storage and then Cloud Volume tab. Click Add Cloud Volume, selecta cloud storage volume (EVS or SFS). If no cloud storage is available, click the linknext to the volume name box to create one. Set Container Path to /var/jenkins_home to mount the volume to the /var/jenkins_home directory of theJenkins container so that Jenkins can retain persistent data.

Figure 14-3 Adding a cloud volume




Step 7 Assign permissions to the Jenkins container so that the Docker commands can berun in the Jenkins container for interconnection with SWR.

1. Enable Privileged Container in Basic Information.

Figure 14-4 Enabling Privileged Container

2. Choose Data Storage > Local Volume, click Add Local Volume, and mountthe host paths /var/run/docker.sock and /usr/bin/docker to thecorresponding container paths. Then, mount the host path /usr/lib64/libltdl.so.7 to the container path /usr/lib/x86_64-linux-gnu/libltdl.so.7.

Figure 14-5 Mounting the host paths to the corresponding container paths

3. In Security Context, set User ID to 0 (user root).

Figure 14-6 Configuring the user




Step 8 Click Next: Set Application Access. Then, click Add Service and configure theworkload access.

Select LoadBalancer or NodePort for Access Type. In this example, selectLoadBalancer.

The Jenkins container image has two ports: 8080 and 50000. You need toconfigure them separately.

● Port 8080 is used for web login. The Service name is jenkins (which can bechanged as required), the container port is 8080, and the access port is 8080.Retain the default values for other parameters.

● Port 50000 is used for the connection between the master and slave nodes.The Service name is slave (which can be changed as required), the containerport is 50000, and the access port is 50000. Retain the default values forother parameters.

Figure 14-7 Adding a Service

Step 9 Click Next: Configure Advanced Settings. Retain the default settings and clickCreate.

Step 10 Click Back to Deployment List to view the Deployment status. If the workload isin Running state, the Jenkins application can be accessed.

Figure 14-8 Viewing the workload status

----End

Logging In and Initializing Jenkins

Step 1 Log in to the CCE console. In the navigation pane, choose Resource Management> Network. On the Services tab page, select the mode for Jenkins to access port8080.




Figure 14-9 Access mode corresponding to port 8080

Step 2 Click the IP address in the Access Address column to open the Jenkinsconfiguration page.

Figure 14-10 Unlocking Jenkins

Step 3 Select the recommended plug-ins.




Figure 14-11 Installing plug-ins

Step 4 Create an administrator.

Figure 14-12 Creating an administrator




Step 5 Configure the instance.

Figure 14-13 Configuring the instance

Figure 14-14 Configuration completed

----End

14.3 Interconnecting Jenkins with SWRDuring Jenkins installation and deployment, Docker commands have beenexecuted in the container. Therefore, no additional configuration is required forJenkins to connect to SWR. You can run the Docker commands to complete thebuild, login, and push operations.




Obtaining a Long-Term Valid Login CommandFor details, see Obtaining a Long-Term Valid Login Command.

For example, the command of this account is as follows:

docker login -u cn-east-3@xxxxx -p xxxxx swr.cn-east-3.myhuaweicloud.com

Create a pipeline to build and push images.The pipeline creation procedure is as follows:

Step 1 Create a pipeline on Jenkins.

Step 2 Configure only the pipeline script.

Pipeline script:node('master'){ stage('Clone') { echo "1.Clone Stage" git url: "https://github.com/lookforstar/jenkins-demo.git"





script { build_tag = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() } } stage('Test') { echo "2.Test Stage" } stage('Build') { echo "3.Build Docker Image Stage" sh "docker build -t swr.cn-east-3.myhuaweicloud.com/batch/jenkins-demo:${build_tag} ." } stage('Push') { echo "4.Push Docker Image Stage" sh "docker login -u cn-east-3@USO32ZAYF95IT3FOE06U -p 5ee790e3ac51bd4e55a8345952e80de563f6d95155a04ab4f5b03ce6fe35a707 swr.cn-east-3.myhuaweicloud.com" sh "docker push swr.cn-east-3.myhuaweicloud.com/batch/jenkins-demo:${build_tag}" } }

Step 3 Save the settings and execute the Jenkins job.

----End

14.4 Interconnecting Jenkins with CCE Clusters

Installing Required Plug-insOn Jenkins, choose System Settings > Plugin Manager. On the Available tabpage, install the Kubernetes Cli Plugin and Kubernetes Continuous DeployPlugin plug-ins.

Plug-in versions:

● Kubernetes Continuous Deploy Plugin: 2.3.0● Kubernetes CLI Plugin: 1.8.3

You can also manually download and install the plug-ins.

http://updates.jenkins-ci.org/download/plugins/kubernetes-cd/

http://updates.jenkins-ci.org/download/plugins/kubernetes-cli/

Adding a Slave Node to JenkinsLog in to Jenkins, choose Manage Jenkins > Manage Nodes, click New Node,and enter the node name slave.

Configure the node information.

Remote working directory: /home/jenkins/agent




http://updates.jenkins-ci.org/download/plugins/kubernetes-cd/

http://updates.jenkins-ci.org/download/plugins/kubernetes-cli/

Boot mode: Launch agent via execution of command on the master

Launch command: docker run -i -u root --rm --name agent -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker -v /usr/lib64/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7 jenkins/agent java -jar /usr/share/jenkins/agent.jar

Configuring and Calling Kubernetes in the Pipeline ScriptImport the kubeconfig configuration file to Jenkins.

Step 1 Create a task to build a freestyle project.

Step 2 Add the Jenkins credential and import the kubeconfig file of the CCE cluster.

Build Environment - Configure Kubernetes CLI (kubectl)




You can obtain the kubeconfig from the CCE cluster details page on HUAWEICLOUD:

Log in to the CCE console. In the navigation pane, choose ResourceManagement > Clusters. Click the name of the cluster where Jenkins is located.On the cluster details page, click the Kubectl tab and download the kubectlconfiguration file.

----End

Creating a Pipeline to Connect to the Kubernetes ClusterFor details about how to use the Kubernetes Continuous Deploy plugin, visit thefollowing website:

https://wiki.jenkins.io/display/JENKINS/Kubernetes+Continuous+Deploy+Plugin

The detailed configuration is as follows:







node('slave'){ stage('Clone') { echo "1.Clone Stage" git url: "https://github.com/lookforstar/jenkins-demo.git" script { build_tag = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() } } stage('Test') { echo "2.Test Stage" } stage('Build') { echo "3.Build Docker Image Stage" sh "docker build -t swr.cn-east-3.myhuaweicloud.com/batch/jenkins-demo:${build_tag} ." } stage('Push') { echo "4.Push Docker Image Stage" sh "docker login -u cn-east-3@82EFQWQMJWSDOYS9BQNM -p 71b0cceac30a10b99a960360b2e1deb04ec2f1bb00e88bc5eb69f3e9fd6a6d08 swr.cn-east-3.myhuaweicloud.com" sh "docker push swr.cn-east-3.myhuaweicloud.com/batch/jenkins-demo:${build_tag}" } stage('Deploy') { echo "5. Deploy Stage" echo "This is a deploy step to test" sh "sed -i 's/<BUILD_TAG>/${build_tag}/' *.yaml" sh "cat *.yaml" echo "begin to config kubenetes" try { kubernetesDeploy( kubeconfigId: "mogujie-spark-zxx", configs: "k8s.yaml") println "hooray, success" } catch (e) { println "oh no! Deployment failed! " println e } }}

Checking the Deployment StatusClick Build Now on Jenkins and view the build result.

Log in to the SWR console and view information about the built image.

Log in to the CCE console and view the workload information.




14.5 Appendix

14.5.1 Complete Pipeline Script for Image Build, Pushing, andDeployment

Example pipeline script:

node('slave'){ stage('Clone') { echo "1.Clone Stage" git url: "https://github.com/lookforstar/jenkins-demo.git" script { build_tag = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() } } stage('Test') { echo "2.Test Stage" } stage('Build') { echo "3.Build Docker Image Stage" sh "docker build -t swr.cn-east-3.myhuaweicloud.com/batch/jenkins-demo:${build_tag} ." } stage('Push') { echo "4.Push Docker Image Stage" sh "docker login -u cn-east-3@82EFQWQMJWSDOYS9BQNM -p 71b0cceac30a10b99a960360b2e1deb04ec2f1bb00e88bc5eb69f3e9fd6a6d08 swr.cn-east-3.myhuaweicloud.com" sh "docker push swr.cn-east-3.myhuaweicloud.com/batch/jenkins-demo:${build_tag}" } stage('Deploy') { echo "5. Deploy Stage" echo "This is a deploy step to test" sh "sed -i 's/<BUILD_TAG>/${build_tag}/' *.yaml" sh "cat *.yaml" echo "begin to config kubenetes" try { kubernetesDeploy( kubeconfigId: "mogujie-spark-zxx", configs: "k8s.yaml") println "hooray, success" } catch (e) { println "oh no! Deployment failed! " println e } }}




14.5.2 Interconnecting Jenkins with RBAC of KubernetesClusters

PrerequisitesRBAC must be enabled for the cluster.

Scenario 1: Namespace-based Permissions ControlCreate a service account and a role, and add a RoleBinding.

$ kubectl create ns dev$ kubectl -n dev create sa dev

$ cat <<EOF > dev-user-role.ymlkind: RoleapiVersion: rbac.authorization.k8s.io/v1metadata: namespace: dev name: dev-user-podrules:- apiGroups: ["*"] resources: ["deployments", "pods", "pods/log"] verbs: ["get", "watch", "list", "update", "create", "delete"]EOFkubectl create -f dev-user-role.yml

$ kubectl create rolebinding dev-view-pod \ --role=dev-user-pod \ --serviceaccount=dev:dev \ --namespace=dev

Generate the kubeconfig file of a specified service account (which can beused for a long time).

$ SECRET=$(kubectl -n dev get sa dev -o go-template='{{range .secrets}}{{.name}}{{end}}')$ API_SERVER="https://172.22.132.51:6443"$ CA_CERT=$(kubectl -n dev get secret ${SECRET} -o yaml | awk '/ca.crt:/{print $2}')$ cat <<EOF > dev.confapiVersion: v1kind: Configclusters:- cluster: certificate-authority-data: $CA_CERT server: $API_SERVER name: clusterEOF

$ TOKEN=$(kubectl -n dev get secret ${SECRET} -o go-template='{{.data.token}}')$ kubectl config set-credentials dev-user \ --token=`echo ${TOKEN} | base64 -d` \ --kubeconfig=dev.conf

$ kubectl config set-context default \ --cluster=cluster \ --user=dev-user \ --kubeconfig=dev.conf

$ kubectl config use-context default \ --kubeconfig=dev.conf

Verification in the CLI

$ kubectl --kubeconfig=dev.conf get poError from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev:dev" cannot list pods in




the namespace "default"

$ kubectl -n dev --kubeconfig=dev.conf run nginx --image nginx --port 80 --restart=Never$ kubectl -n dev --kubeconfig=dev.conf get poNAME READY STATUS RESTARTS AGEnginx 1/1 Running 0 39s

Verify whether the permissions meet the expectation in Jenkins.

Step 1 Add the kubeconfig file with permissions control settings to Jenkins.

Step 2 Start the Jenkins job. In this example, Jenkins fails to be deployed in namespacedefault but is successfully deployed in namespace dev.

----End

Scenario 2: Resource-based Permissions Control

Step 1 Generate the service account, role, and binding. kubectl -n dev create sa sa-test0304

cat <<EOF > test0304-role.ymlkind: RoleapiVersion: rbac.authorization.k8s.io/v1metadata: namespace: dev




name: role-test0304rules:- apiGroups: ["*"] resources: ["deployments"] resourceNames: ["tomcat03", "tomcat04"] verbs: ["get", "update", "patch"]EOFkubectl create -f test0304-role.yml

kubectl create rolebinding test0304-bind \ --role=role-test0304 \ --serviceaccount=dev:sa-test0304\ --namespace=dev

Step 2 Generate the kubeconfig file. SECRET=$(kubectl -n dev get sa sa-test0304 -o go-template='{{range .secrets}}{{.name}}{{end}}') API_SERVER=" https://192.168.0.153:5443" CA_CERT=$(kubectl -n dev get secret ${SECRET} -o yaml | awk '/ca.crt:/{print $2}') cat <<EOF > test0304.confapiVersion: v1kind: Configclusters:- cluster: certificate-authority-data: $CA_CERT server: $API_SERVER name: clusterEOF

TOKEN=$(kubectl -n dev get secret ${SECRET} -o go-template='{{.data.token}}') kubectl config set-credentials test0304-user \ --token=`echo ${TOKEN} | base64 -d` \ --kubeconfig=test0304.conf

kubectl config set-context default \ --cluster=cluster \ --user=test0304-user \ --kubeconfig=test0304.conf

kubectl config use-context default \ --kubeconfig=test0304.conf

Step 3 Verify that Jenkins is running as expected.

In the pipeline script, update the Deployments of tomcat03, tomcat04, andtomcat05 in sequence.

try { kubernetesDeploy( kubeconfigId: "test0304", configs: "test03.yaml") println "hooray, success" } catch (e) { println "oh no! Deployment failed! " println e } echo "test04" try { kubernetesDeploy( kubeconfigId: "test0304", configs: "test04.yaml") println "hooray, success" } catch (e) { println "oh no! Deployment failed! " println e } echo "test05" try { kubernetesDeploy( kubeconfigId: "test0304",




configs: "test05.yaml") println "hooray, success" } catch (e) { println "oh no! Deployment failed! " println e }

Viewing the running result:

Figure 14-15 test03




Figure 14-16 test04

----End

14.5.3 Publishing an HTTPS Ingress to ELB

Obtaining and Encrypting the ELB Certificate

Step 1 Obtain the example ELB certificate and private key.

Log in to the management console and click in the upper left corner. In theservice list, choose Network > Elastic Load Balance. In the navigation pane,choose Elastic Load Balance > Certificates. On the page displayed, click CreateCertificate.

In the dialog box displayed, configure the certificate and private key. In thissection, the example values are used. You can click next to View Example toautomatically populate the values.




Figure 14-17 Obtaining the certificate and private key

Step 2 Write the certificate and private key to two files and use cat xxx | base64 | tr -d'\n' to perform Base64 encryption to obtain the encrypted fields.

Figure 14-18 Obtaining the encrypted fields

----End

Generating the Secret FileThe generated secret file is as follows:

apiVersion: v1data: tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0******dDOTlIV3A2S3czN1JMOFdvQjhHV0ZVMFE0dEhMT2pCSXhrWlJPUFJoSCt6TUlycVVleHY2ZnNiM05XS2hubGZoMU1qNXdRRTRMZG89Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRV*******SGp5YTRJa2trZ3Q0MHExTXJVc2dJWWJGWU1mMgotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=kind: Secretmetadata: annotations: description: test for ingressTLS secrets name: elb-https-secret namespace: defaulttype: IngressTLS

Create the secret.

kubectl create –f xxx.yml

Create an ingress and use the secret.apiVersion: extensions/v1beta1kind: Ingressmetadata: annotations:




kubernetes.io/elb.ip: 192.168.0.244 kubernetes.io/elb.port: "7777" name: https-ing namespace: defaultspec: rules: - http: paths: - backend: serviceName: jenkinstest servicePort: 8080 path: /test-https property: ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH tls: - secretName: elb-https-secret

Create an ingress.

kubectl create –f xxx.yml




Best Practice - Huawei...2.9 Step 6: Upgrading the Game Application.....44 2.10 Step 7: Deleting...

Documents

Transcript of Best Practice - Huawei...2.9 Step 6: Upgrading the Game Application.....44 2.10 Step 7: Deleting...