How to Architect and Implement Devopson ExaCS€¦ · Domain 1 Availability Domain 2 Availability...

Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|

HowtoArchitectandImplementDevops onExaCSTIP4120

JeffreyT.WrightSr.PrincipalProductManagerDatabaseCloudServices,ExadataOctober25,2018

Copyright©2018,Oracleand/oritsaffiliates.Allrightsreserved.|

Agenda

• Architectureblockdiagram• Systemcredentials• OCITerraformtodeployExaCS,scaleExaCSOCPUs• OCIPythonSDKforuserandgroupadministration• DatabasecreationandpatchingviaPython• Demo

2

Copyright©2018,Oracleand/oritsaffiliates.Allrightsreserved.| 3

OCIArchitectureBlockDiagram

Tenancy

Region– e.g.,FrankfurtAvailabilityDomain1

AvailabilityDomain2

AvailabilityDomain3

Compartment

ExaCS

ComputeVCN

ExaCS

Compute

ExaCS

Compute

Region– e.g.,PhoenixAvailabilityDomain1

AvailabilityDomain2

AvailabilityDomain3

Compartment

ExaCS

ComputeVCN

ExaCS

Compute

ExaCS

Compute


OCIArchitectureBlockDiagram

Tenancy


AvailabilityDomain2

AvailabilityDomain3

CompartmentVCN

Region– e.g.,PhoenixAvailabilityDomain1

AvailabilityDomain2

AvailabilityDomain3

ExaCS

Compute

ExaCS

Compute

ExaCS

Compute

ExaCS

Compute

ExaCS

Compute

ExaCS

Compute


ExadataCloudServiceExampleDBSystem

Tenancy


CompartmentVCN

ExaCS

ClientSubnetBackupSubnet

InternetGatewayServiceGateway

RoutingRules

SecurityLists


SystemCredentials

• Cloudcredentials– Giveyouaccesstocloudautomationtodeploycloudresources– welcomeemail

• VirtualMachinecredentials– Giveyouaccesstothevirtualmachinesyoucreatewithyourcloudcredentials– Tokenbasedssh – installedviacloudautomation

• DatabaseServicecredentials– Giveyouaccesstodatabasesyoucreatewithyourcloudcredentials– Passwordsspecifiedwhenyoucreatethedatabaseservice

6

Howtologin


WelcomeEmail


OCIConsole


SettingupyourCredentials• KeysandOCIDs– https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm

• MakeanRSApublic/privatekeypairinPEMformat• GetthefingerprintofyourRSAkey• GetyourtenancyOCIDanduserOCID• Setupyourlocalenvironmentvariablestorunterraform– https://www.terraform.io/docs/providers/oci/

9


MaketheKeys# make the private keyopenssl genrsa -out ~/.oci/oci_api_key.pem 2048chmod go-rwx ~/.oci/oci_api_key.pem

# make the public key from the private keyopenssl rsa -pubout -in ~/.oci/oci_api_key.pem -out ~/.oci/oci_api_key_public.pem

# copy the public key to the clipboardcat ~/.oci/oci_api_key_public.pem | pbcopy

# get the key's fingerprintopenssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem | openssl md5 -c

10


GettheTenancyOCID

11


GettheTenancyOCID

12


GettheUserOCID

13


GettheUserOCID

14


AddYourPublicKey

15


GetYourCompartmentOCID


EnvironmentVariablesexport OCI_GO_SDK_DEBUG=1export TF_LOG=DEBUG

export TF_VAR_tenancy_ocid=ocid1.tenancy.oc1..aaaaaaaambnyexdtahy6ug7dy2ngnfnthvvbpfgmgmg3slb73f52wkbudvwqexport TF_VAR_region=eu-frankfurt-1

export TF_VAR_compartment_ocid=ocid1.compartment.oc1..aaaaaaaarimctbi7gbn45dj4zdpvuny75aa6lajw2vbvzepgioydyv2i57pq

export TF_VAR_user_ocid=ocid1.user.oc1..aaaaaaaagzwawhkatr2yd74nnox4hpylpllvbau7t3wzehqk7nynpmkhbbcqexport TF_VAR_private_key_path=/Users/jtwright/.oci/oci_api_key.pemexport TF_VAR_fingerprint=a0:fc:3c:2c:e1:9c:4c:3b:7b:37:b7:4a:12:42:ef:56

export TF_VAR_vcn=ocid1.vcn.oc1.eu-frankfurt-1.aaaaaaaaleafo2w7glqy4sarriqlaj7utqz2o4oeyxpmryjw7kg65dxgjnfq

18


GetTerraformandtheOCIProvider(Plugin)• Formaldoc– https://www.terraform.io/docs/providers/oci/

• Terraformdownload– https://github.com/hashicorp/terraform• https://github.com/hashicorp/terraform/archive/master.zip

• OCIProviderdownload– https://github.com/terraform-providers/terraform-provider-oci• https://github.com/terraform-providers/terraform-provider-oci/archive/master.zip

19


Non-DatabaseSystemVariableSetupvariable "tenancy_ocid" {}variable "user_ocid" {}variable "fingerprint" {}variable "private_key_path" {}variable "compartment_ocid" {}variable "region" {}provider "oci" {

tenancy_ocid = "${var.tenancy_ocid}"user_ocid = "${var.user_ocid}"fingerprint = "${var.fingerprint}"private_key_path = "${var.private_key_path}"region = "${var.region}"

}variable "CLIENT-CIDR" { default = "10.10.3.0/24" }variable "BACKUP-CIDR" { default = "10.10.4.0/24" }variable "VPC-CIDR" { default = "10.10.0.0/16" }variable "availability_domain" { default = "2" }data "oci_identity_availability_domains" "ADs" {

compartment_id = "${var.tenancy_ocid}"}data "oci_core_services" "test_services" {

filter {name = "name"values = [".*Object.*Storage"]regex = true

}}

20


DatabaseSystemVariableSetupvariable "db_disk_redundancy" { default = "HIGH" }variable "hostname" { default = "exanode" }variable "db_edition" { default = "ENTERPRISE_EDITION_EXTREME_PERFORMANCE" }variable "db_admin_password" { default = "WelcoMe-OCI#123" }variable "db_name" { default = "TESTDB" }variable "character_set" { default = "AL32UTF8" }variable "n_character_set" { default = "AL16UTF16" }variable "db_workload" { default = "DSS" }variable "pdb_name" { default = "TESTPDB" }variable "db_version" { default = "12.2.0.1" }variable "db_home_display_name" { default = "TESTDBHOME" }variable "db_system_shape" { default = "Exadata.Quarter1.84" }variable "cpu_core_count" { default = "22" }variable "data_storage_percentage" { default = "80" }variable "db_system_display_name" { default = "TESTDB" }variable "license_model" { default = "LICENSE_INCLUDED" }

variable "ssh_public_key" { default = "ssh-rsaAAAAB3NzaC1yc2EAAAADAQABAAABAQCaqkoNWcD3QDPH4H6LvUizCwcO1Gnk8Xke2b8VToK8U9PjwGDYdH6afdkF2fq0u+kpbiW15AJoJgSvUKIORn+L+htYF9aWRJ3DcAjm9xaSpb+aFPxeh0b5WbYyL5kISD/uBDJvauubIHe9P1ccHv82Tl1EOVrwL0S3N6wFIOCza4ZGXvpyOM9eKyuxy3qnmQDGJ+SKDzT/Yql4NthIhIMcU4IXUqixwpUf5q8PuZzqdbkMaeIHdkiXh08y1YfOu60ypilWaVBEGPC1PJ3hqnE4ZBZy4BHkushuhIOVjP+QodmjaGStlJtvaabGJOMXoOeKzFhNq26qLUW9G2eIcrct jtwright@jtwright-Mac" }

21


CreateandExaCSinOCIin8Steps

1. Getyouruserid andcompartmentfromyouradministrator2. CreateaVCN3. CreateGateways4. CreateRouteTables5. CreateSecurityLists6. CreateDHCPOptions7. CreateSubnets8. LaunchanExadataDBSystem

22

https://cloud.oracle.com/iaas/whitepapers/Deploying_Exadata_on_OCI.pdf


VCNandGatewaysresource "oci_core_vcn" "ExaVCN" {cidr_block = "${var.VPC-CIDR}"compartment_id = "${var.compartment_ocid}"dns_label = "frankfurt"display_name = "frankfurt"

}

resource "oci_core_internet_gateway" "ExaIG" {compartment_id = "${var.compartment_ocid}"display_name = "ExaIG"vcn_id = "${oci_core_vcn.ExaVCN.id}"

}

resource "oci_core_service_gateway" "ExaSG" {compartment_id = "${var.compartment_ocid}"services {

service_id = "${lookup(data.oci_core_services.test_services.services[0], "id")}"}

display_name = "ExaSG"vcn_id = "${oci_core_vcn.ExaVCN.id}"

}

23


RouteTablesresource "oci_core_route_table" "Client_RT" {compartment_id = "${var.compartment_ocid}"vcn_id = "${oci_core_vcn.ExaVCN.id}"display_name = "Client_RT"

route_rules {cidr_block = "0.0.0.0/0"network_entity_id = "${oci_core_internet_gateway.ExaIG.id}"

}}

resource "oci_core_route_table" "Backup_RT" {compartment_id = "${var.compartment_ocid}"vcn_id = "${oci_core_vcn.ExaVCN.id}"display_name = "Backup_RT"

route_rules {destination = "${lookup(data.oci_core_services.test_services.services[0],

"cidr_block")}"destination_type = "SERVICE_CIDR_BLOCK"network_entity_id = "${oci_core_service_gateway.ExaSG.id}"}

}

24


NodeTrafficSecurityListresource "oci_core_security_list" "NodeTraffic" { compartment_id = "${var.compartment_ocid}" display_name = "NodeTraffic" vcn_id = "${oci_core_vcn.ExaVCN.id}"

egress_security_rules = [{ destination = "${var.CLIENT-CIDR}" protocol = "6" }, { destination = "${var.BACKUP-CIDR}" protocol = 6 }, { destination = "${var.CLIENT-CIDR}" protocol = 1 }, { destination = "${var.BACKUP-CIDR}" protocol = 1 } ]

ingress_security_rules = [{ source = "${var.CLIENT-CIDR}" protocol = "6" }, { source = "${var.BACKUP-CIDR}" protocol = 6 }, { source = "${var.CLIENT-CIDR}" protocol = 1 }, { source = "${var.BACKUP-CIDR}" protocol = 1 } ]}

25


SSHTrafficSecurityListresource "oci_core_security_list" "SSHTraffic" {compartment_id = "${var.compartment_ocid}"display_name = "SSHTraffic"vcn_id = "${oci_core_vcn.ExaVCN.id}"

ingress_security_rules {protocol = "6" // tcpsource = "0.0.0.0/0"stateless = falsetcp_options = {"min" = 22"max" = 22

}}

}

26


SQLNet TrafficSecurityListresource "oci_core_security_list" "SQLNet" { compartment_id = "${var.compartment_ocid}" display_name = "SQLNet" vcn_id = "${oci_core_vcn.ExaVCN.id}"

ingress_security_rules = [{ protocol = "6" source = "${var.CLIENT-CIDR}" tcp_options = { "min" = 1521 "max" = 1521 }

}, { protocol = "6" source = "${var.BACKUP-CIDR}" tcp_options = { "min" = 1521 "max" = 1521 } } ]}

27


ClientTrafficSecurityListresource "oci_core_security_list" "Client" { compartment_id = "${var.compartment_ocid}" display_name = "Client" vcn_id = "${oci_core_vcn.ExaVCN.id}"

egress_security_rules = [{ destination = "${var.CLIENT-CIDR}" protocol = "6" }, { destination = "${var.BACKUP-CIDR}" protocol = 6 }, { destination = "${var.CLIENT-CIDR}" protocol = 1 }, { destination = "${var.BACKUP-CIDR}" protocol = 1 }, ]

ingress_security_rules = [{ source = "${var.CLIENT-CIDR}" protocol = "6" }, { source = "${var.BACKUP-CIDR}" protocol = 6 }, { source = "${var.CLIENT-CIDR}" protocol = 1 }, { source = "${var.BACKUP-CIDR}" protocol = 1 }, ]}

28


BackupTrafficSecurityListresource "oci_core_security_list" "DB_Backup" {

compartment_id = "${var.compartment_ocid}"display_name = "DB_Backup"vcn_id = "${oci_core_vcn.ExaVCN.id}"

egress_security_rules = [{destination = "${var.CLIENT-CIDR}"protocol = "6"

},{destination = "${var.BACKUP-CIDR}"protocol = 6

}]

ingress_security_rules = [{protocol = "6"source = "${var.CLIENT-CIDR}"

},{protocol = "6"source = "${var.BACKUP-CIDR}"

}]

}

29


EmptySecurityListsforFutureUseresource "oci_core_security_list" "Flex1" {compartment_id = "${var.compartment_ocid}"display_name = "Flex1"vcn_id = "${oci_core_vcn.ExaVCN.id}"

# empty rule set}

resource "oci_core_security_list" "Flex2" {compartment_id = "${var.compartment_ocid}"display_name = "Flex2"vcn_id = "${oci_core_vcn.ExaVCN.id}"

# empty rule set}

30


DHCPOptionsresource "oci_core_dhcp_options" "ExaDHCP" {compartment_id = "${var.compartment_ocid}"vcn_id = "${oci_core_vcn.ExaVCN.id}"display_name = "ExaDHCP"

options {type = "DomainNameServer"server_type = "VcnLocalPlusInternet"

}}

31


ClientSubnetresource "oci_core_subnet" "Client_Subnet_AD2" {availability_domain =

"${lookup(data.oci_identity_availability_domains.ADs.availability_domains[var.availability_domain - 1],"name")}"cidr_block = "${var.CLIENT-CIDR}"display_name = "Client_Subnet_AD2"compartment_id = "${var.compartment_ocid}"vcn_id = "${oci_core_vcn.ExaVCN.id}"route_table_id = "${oci_core_route_table.Client_RT.id}"security_list_ids = ["${oci_core_security_list.Client.id}",

"${oci_core_security_list.SQLNet.id}","${oci_core_security_list.Flex1.id}","${oci_core_security_list.NodeTraffic.id}","${oci_core_security_list.SSHTraffic.id}"

]dns_label = "clientad2"dhcp_options_id = "${oci_core_dhcp_options.ExaDHCP.id}"

}

32


BackupSubnetresource "oci_core_subnet" "Backup_Subnet_AD2" {availability_domain =

"${lookup(data.oci_identity_availability_domains.ADs.availability_domains[var.availability_domain - 1],"name")}"cidr_block = "${var.BACKUP-CIDR}"display_name = "Backup_Subnet_AD2"compartment_id = "${var.compartment_ocid}"vcn_id = "${oci_core_vcn.ExaVCN.id}"route_table_id = "${oci_core_route_table.Backup_RT.id}"dns_label = "backupad1"security_list_ids = ["${oci_core_security_list.NodeTraffic.id}",

"${oci_core_security_list.DB_Backup.id}","${oci_core_security_list.SSHTraffic.id}","${oci_core_security_list.Flex2.id}"

]dhcp_options_id = "${oci_core_dhcp_options.ExaDHCP.id}"

}

33


CreatetheExadataDBSystemresource "oci_database_db_system" "DemoExaCS" {availability_domain = "${lookup(data.oci_identity_availability_domains.ADs.availability_domains[var.availability_domain - 1],"name")}"compartment_id = "${var.compartment_ocid}"database_edition = "${var.db_edition}"cpu_core_count = "${var.cpu_core_count}"

db_home {database {admin_password = "${var.db_admin_password}"db_name = "${var.db_name}"character_set = "${var.character_set}"ncharacter_set = "${var.n_character_set}"db_workload = "${var.db_workload}"pdb_name = "${var.pdb_name}"

db_backup_config {auto_backup_enabled = true

}}

db_version = "${var.db_version}"display_name = "${var.db_home_display_name}"

}

shape = "${var.db_system_shape}"subnet_id = "${oci_core_subnet.Client_Subnet_AD2.id}"backup_subnet_id = "${oci_core_subnet.Backup_Subnet_AD2.id}"ssh_public_keys = ["${var.ssh_public_key}"]display_name = "${var.db_system_display_name}"

hostname = "${var.hostname}"data_storage_percentage = "${var.data_storage_percentage}"license_model = "${var.license_model}"

}

34


PlanandApply!jtwright-Mac:jtw jtwright$ pwd/Users/jtwright/.terraform.d/plugins/terraform-provider-oci-master/docs/examples/jtwjtwright-Mac:jtw jtwright$ terraform plan...Plan: 16 to add, 0 to change, 1 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraformcan't guarantee that exactly these actions will be performed if"terraform apply" is subsequently run.

jtwright-Mac:jtw jtwright$ terraform apply...

Plan: 16 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?Terraform will perform the actions described above.Only 'yes' will be accepted to approve.

Enter a value: yes...Apply complete! Resources: 16 added, 0 changed, 1 destroyed.

35


HowtoScaleOCPUs

36

SimplychangetheDBSystemvariableforOCPUcount,plan,andapplyvariable "cpu_core_count" { default = ”44" }...

Terraform will perform the following actions:

~ oci_core_dhcp_options.ExaDHCP2018/10/19 13:43:52 [DEBUG] plugin: waiting for all plugin processes to complete...

options.#: "2" => "1"options.1.search_domain_names.#: "1" => "0"options.1.search_domain_names.0: "frankfurt.oraclevcn.com" => ""options.1.type: "SearchDomain" => ""

~ oci_database_db_system.DemoExaCScpu_core_count: "22" => "44"

Plan: 0 to add, 2 to change, 0 to destroy....Apply complete! Resources: 0 added, 2 changed, 0 destroyed.jtwright-Mac:jtw jtwright$


DemoExadataDBSysteminOCIviaTerraform

37


PythonAPISDK

38

https://docs.cloud.oracle.com/iaas/Content/API/SDKDocs/pythonsdk.htm

#!/usr/bin/pythonimport oci, pprint, osfrom oci.identity.models import AddUserToGroupDetailsfrom oci.config import from_file

# config = from_file()

config = { "compartment" : ocid1.compartment.oc1..aaaaaaaarimctbi7gbn45dj4zdpvuny75aa6lajw2vbvzepgioydyv2i57pq,

"region": eu-frankfurt-1,"tenancy" : ocid1.tenancy.oc1..aaaaaaaambnyexdtahy6ug7dy2ngnfnthvvbpfgmgmg3slb73f52wkbudvwq,"user" : ocid1.user.oc1..aaaaaaaagzwawhkatr2yd74nnox4hpylpllvbau7t3wzehqk7nynpmkhbbcq,"fingerprint": a0:fc:3c:2c:e1:9c:4c:3b:7b:37:b7:4a:12:42:ef:56,"key_file": /Users/jtwright/.oci/oci_api_key.pem,"vcn" : ocid1.vcn.oc1.eu-frankfurt-1.aaaaaaaaleafo2w7glqy4sarriqlaj7utqz2o4oeyxpmryjw7kg65dxgjnfq

}

pprint.pprint( config )

identity = oci.identity.IdentityClient(config)compartment_id = config["tenancy"]


CreateaGroupandaUser

39

from oci.identity.models import CreateGroupDetailsrequest = CreateGroupDetails()request.compartment_id = compartment_idrequest.name = "exa-pm-group"request.description = "Exadata PM group"

group = identity.create_group(request)

pprint.pprint( group.data )

request = CreateUserDetails()request.compartment_id = compartment_idrequest.name = "exadata-pm"request.description = "Created with the Python SDK"user = identity.create_user(request)

pprint.pprint(user.data)

request = AddUserToGroupDetails()request.group_id = group.data.idrequest.user_id = user.data.idresponse = identity.add_user_to_group(request)

pprint.pprint(response.data)


CreateaDatabase

40

ExadataCloudatCustomer,OracleCloudInfrastructureClassicx_id_tenant_name = 'a516946'user = '[email protected]'password = 'Welc0me!'baseurl='https://dbaas.oraclecloud.com/’servicesurl='paas/service/dbcs/api/v1.1/instances/'url = baseurl + servicesurl + x_id_tenant_nameheaders = {'X-ID-TENANT-NAME':x_id_tenant_name, 'content-type': 'application/json'}

payload = { "serviceName" : "JTWTEST","description" : "Test database for JTW","edition" : "EE_EP","exadataSystemName": "oow2017exa","level" : "PAAS_EXADATA","subscriptionType": "MONTHLY","version": "12.1.0.2","vmPublicKeyText": "ssh-rsa

AAAAB3NzaC1yc2EAAAADAQABAAABAQCaqkoNWcD3QDPH4H6LvUizCwcO1Gnk8Xke2b8VToK8U9PjwGDYdH6afdkF2fq0u+kpbiW15AJoJgSvUKIORn+L+htYF9aWRJ3DcAjm9xaSpb+aFPxeh0b5WbYyL5kISD/uBDJvauubIHe9P1ccHv82Tl1EOVrwL0S3N6wFIOCza4ZGXvpyOM9eKyuxy3qnmQDGJ+SKDzT/Yql4NthIhIMcU4IXUqixwpUf5q8PuZzqdbkMaeIHdkiXh08y1YfOu60ypilWaVBEGPC1PJ3hqnE4ZBZy4BHkushuhIOVjP+QodmjaGStlJtvaabGJOMXoOeKzFhNq26qLUW9G2eIcrct jtwright@jtwright-Mac",

"parameters": [{

"adminPassword": "Welc0me#1","backupDestination": "none","isRac": "yes","pdbName": "JTWTESTPDB","sid": "JTWTEST","type": "db"

}]}

response = requests.post( url, headers = headers, auth = (user, password), data=json.dumps( payload ) )


PatchaDatabase

41

ExadataCloudatCustomer,OracleCloudInfrastructureClassicservicesurl="/paas/api/v.1/instancemgmt/"url = baseurl + servicesurl + x_id_tenant_name + "/services/dbaas/instances/" + "JTWTEST" + "/patches/available"headers = {'X-ID-TENANT-NAME':x_id_tenant_name}response = requests.get( url, headers = headers, auth = (user, password) )

newestPatchId = 0for data in response.json():

if data["serviceType"] == "DBaaS":print data["releaseDate"] + " " + data["patchNumber"] + " " + data["patchId"]if data["patchId"] > newestPatchId:

newestPatchId = data["patchId"]

servicesurl="/paas/api/v1.1/instancemgmt/"url = baseurl + servicesurl + x_id_tenant_name + "/services/dbaas/instances/" + "JTWTEST" + "/patches/checks/" + newestPatchIdheaders = {'X-ID-TENANT-NAME':x_id_tenant_name, 'content-type': 'application/json'}

payload = { "additionalNote" : "Patch precheck using REST API" }response = requests.put( url, headers = headers, auth = (user, password), data=json.dumps( payload ) )

jobId=response.json()["details"]["jobId"]servicesurl="/paas/api/v1.1/instancemgmt/"url = baseurl + servicesurl + x_id_tenant_name + "/services/dbaas/instances/" + "JTWTEST" + "/patches/checks/job/" + jobIdheaders = {'X-ID-TENANT-NAME':x_id_tenant_name}

while True:response = requests.get( url, headers = headers, auth = (user, password) )status=response.json()["status"]print statusif status != "IN_PROGRESS":

breaktime.sleep( 10 )


PatchaDatabase

42

ExadataCloudatCustomer,OracleCloudInfrastructureClassicservicesurl="/paas/api/v1.1/instancemgmt/"url = baseurl + servicesurl + x_id_tenant_name + "/services/dbaas/instances/" + "JTWTEST" + "/patches/checks/" + newestPatchIdheaders = {'X-ID-TENANT-NAME':x_id_tenant_name, 'content-type': 'application/json'}

payload = { "additionalNote" : "Patch precheck using REST API" }response = requests.put( url, headers = headers, auth = (user, password), data=json.dumps( payload ) )

jobId=response.json()["details"]["jobId"]servicesurl="/paas/api/v1.1/instancemgmt/"url = baseurl + servicesurl + x_id_tenant_name + "/services/dbaas/instances/" + "JTWTEST" + "/patches/checks/job/" + jobIdheaders = {'X-ID-TENANT-NAME':x_id_tenant_name}

while True:response = requests.get( url, headers = headers, auth = (user, password) )status=response.json()["status"]print statusif status != "IN_PROGRESS":

breaktime.sleep( 10 )

How to Architect and Implement Devopson ExaCS€¦ · Domain 1 Availability Domain 2 Availability...

Documents

Transcript of How to Architect and Implement Devopson ExaCS€¦ · Domain 1 Availability Domain 2 Availability...