As a Service: Cloud Foundry on OpenStack - Lessons Learnt

38
As a Service: Cloud Foundry on OpenStack - Lessons Learnt Apps @AnimeshSingh OpenStack Summit Barcelona October 2016 @blueboxjesse

Transcript of As a Service: Cloud Foundry on OpenStack - Lessons Learnt

As a Service: Cloud Foundry on OpenStack - Lessons Learnt

Apps

@AnimeshSingh

OpenStack Summit Barcelona

October 2016 @blueboxjesse

A polyglot “platform for the people”

•  The de facto open PaaS platform

•  Foundation established Dec. 2014, under Linux Foundation umbrella

Cloud Foundry

Cloud Foundry Community

60+ member companies driving Cloud Foundry.

PaaS

Cloud Foundry on OpenStack

IaaS

UAA

Router

DEAPool

ServiceGateway Apps

ServiceConnector

HealthManager

Messaging

CloudController

BuildPacks

ServiceNodes

CloudFoundryBOSH

CloudProviderInterface

BOSH Deployment Process

Deployment Manifest •  Release name/version •  # VMs, job params •  Stemcells to use

Stemcell •  Base OS •  BOSH agent

Release •  Name •  Software packages •  Config templates •  Scripts

BOSH

Cloud Foundry

Virtual Machine •  Configuration •  Software Packages

Virtual Machine •  Configuration •  Software Packages

Virtual Machine •  Configuration •  Software Packages

Virtual Machine •  Configuration •  Software packages

© IBM Corporation 6

Problems: Faced, by us and by the community!

Problems we faced As CF and OpenStack deployers we experienced various issues deploying CF and OpenStack, and CF on OpenStack

§  Instability: Instability of OpenStack environments from various distros

§  APIs: Lack of compliance in APIs and new releases. Some of the API behavior also differs based on the plugins used

§  Capacity: Right from cpu/mem/disk etc. to HA, persistent disk, floating ips etc. – sizing needs to be precise

§  Network: Should CF co reside with other management components in a network?

§  OpenStack for Enterprise Software: Enterprise software lags, with most variations still available only for VMware.

§  Generic Deployment: Consistency when deploying CF on OpenStack, VMware or any other IaaS §  Combined OpenStack and Cloud Foundry Usage: How to allow seamless usage and consumption of OpenStack

services along with CF services at the same time?

§  CF HA: What works, what doesn’t?

§  Proxy: Environments behind customer firewalls with restricted outbound access §  Constant Release Cycles: Both CF and OpenStack have frequent releases, patches, updates etc. How do we

reconcile?

7

50% experience significant issues deploying CF onto OS

45% are intermediate OS users

Problems Community Faces: Survey Results

Close to 50% of users had to customize their environment

for CF

Problems Community Faces: Survey Results

Most users on Juno/Kilo Close to 50% of users run their own local OpenStack

Problems Community Faces: Survey Results

© IBM Corporation 11

Our OpenStack and Cloud Foundry Offerings

APACHE HAPROXY APACHE HAPROXY

HA MYSQL ARBITER

(PERCONA)

HA NEUTRON !

HA KEYSTONE!

HA GLANCE!

HA HORIZON!

NOVA!

CINDER API & ISCSI!

HA NEUTRON !

HA KEYSTONE!

HA GLANCE!

HA HORIZON!

NOVA!

CINDER API & ISCSI!

HA MYSQL (PERCONA) RABBITMQ HA MYSQL

(PERCONA) RABBITMQ

Our OpenStack Offering - Bluemix Private Cloud (IaaS)

NOVA

CINDER API & ISCSI

NOVA

CINDER API & ISCSI

§  IBM Platform as a Services offering •  Three deployments styles: public, dedicated, private •  1M+ registered users (20K+/month) •  100K+ running apps and 500+ services

Bluemix Bare Metal (public and dedicated) and OpenStack/VMware (private)

Our Cloud Foundry Offering - Bluemix (PaaS)

Services

Lifecycle Management

IDS

Application Runtime

Runtimes & Frameworks

Middleware Application Operational Mobile External Data

Node Java Ruby Worklight WebSphere Liberty

Eclipse IDE Application

Composition Environment

Create & Manage Services

Test/Run Test/Run

Explore Services

Explore Services

IBM Bluemix

Check In Code Check In Code

Web IDE (Eclipse Orion)

© IBM Corporation 14

Lessons Learnt deploying Cloud Foundry on OpenStack

Test if your OpenStack is fit for Cloud Foundry deployment

Manually, you can run the validations here to see if your OpenStack is a good fit: https://docs.cloudfoundry.org/deploying/openstack/validate_openstack.html §  Can you access the OpenStack APIs for your instance of OpenStack?

§  Can you access OpenStack metadata service from a virtual machine?

§  Can you ping one virtual machine from another?

§  Can you invoke large numbers of API calls?

§  Can you create and mount a large volume?

§  Can you upload and deploy an Ubuntu Server Cloud Image?

§  Can networking be configured for both external and internal IPs?

§  Can you access the Internet from within instances?

Test if your OpenStack is fit for Cloud Foundry deployment To check in automated fashion if your OpenStack is ready to run BOSH and Cloud Foundry, you can run this CF Incubator project : https://github.com/cloudfoundry-incubator/cf-openstack-validator

16

Testing the CPI API Upload stemcell Create VM Find VM Create disk Find disk Attach disk to VM Detach disk from VM Create disk snapshot Delete disk snapshot Delete disk Delete VM Delete stemcell

Other OpenStack tests Check API rate limit Check required versions of OpenStack projects CPI requires API version 1 for glance and cinder Security group settings Check if security group rules allow necessary incoming/outgoing ports Outbound internet access Timeservers can be reached Attach a floating IP Set VM metadata tags Access a VM over ssh from the outside Access one VM from another VM Create a large volume

Get the right sizing for Cloud Foundry •  Sample sizing are available online as references e.g

https://docs.cloudfoundry.org/deploying/openstack/required-flavors.html https://www.mirantis.com/blog/full-stack-devops-with-pivotal-cloud-foundry-on-mirantis-openstack https://docs.pivotal.io/pivotalcf/1-7/customizing/openstack.html#versions

•  Increase default quota of the tenant to meet minimum requirement e.g number of ports always gets hit first •  Recommended ‘OpenStack flavors’ disk space should be go in ephemeral disk.

Keep root disk to 10GB minimum

Optimize OpenStack scheduling and communication

Optimize Internal Communication: •  Configure OpenStack for high volume API calls e.g Increase OpenStack API rate limits (/etc/nova/api-paste.ini) •  Avoid name based security groups with nova-network. Name

based security groups require message bus activity and database updates proportional to the number of existing VMs

•  If Neutron is configured with VXLAN via the Open vSwitch mechanism, the MTU should be 1400. For GRE, the recommended number is 1460.

Use the right Scheduler

Use an OpenStack scheduler which evenly distributes load e.g compute_scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler

Account for different OpenStack configurations •  If your OpenStack setup requires you to use disks from block

storage instead, that will work with Cloud Foundry as well. properties. openstack.boot_from_volume: true

•  By default, the VMs created try to receive data from OpenStack's HTTP metadata service. If your OpenStack installation doesn't provide metadata and userdata over HTTP, but requires you to use config-drive instead of metadata, you need to specify this in the property properties.openstack.config_drive: cdrom

Optimize Cloud Foundry / BOSH bandwidth and communication

Optimize Internal Communication: •  Increase BOSH NATS time out for high concurrency

e.g Configure NATS messaging bus to increase ping interval

Optimized routing and bandwidth allocation •  Isolate Cloud Foundry components using multiple networks

e.g DEAs in their own network, brokered services in their own network

Optimize log forwarding

•  Supporting services like logging, report generation etc should go in their own tenant network. Also any communication between the VM(s) sending logs, and log receiving component(s) should go over the private network. Don’t use floating ips as destination, else you are paying double the cost.

Optimize for Security and Network

•  Only open ports which are needed. Use the most limited permissions required to complete the job.

•  If OpenStack is using a self-signed certificate, configure properties.openstack.connection_options to include the property ca_cert

•  Use tenant credentials: Do not use full admin credentials in your BOSH manifest

•  Minimize floating ips: Except for the incoming Gateway device(HA Proxy, Datapower, F5 etc), none of the fabric VMs should be on public network or need a floating ip

Account for customer boundary firewall or proxy •  BOSH will retrieve CF release components via the

URL provided in deployment manifest. Also services and CF apps may need outbound access. In environments behind firewall or proxy with restricted access, this could be a major issue.

•  If the firewall or proxy requires both destination and source ips to be enlisted, prepare a list of destination IPs/URLs you need to reach out and hand it to datacenter admin

•  If your OpenStack VMs don’t have a floating ip from external network, the source ip presented to firewall will be Neutron gateway ip

•  Cloud Foundry doesn’t support out of the box SSL packet inspection where the SSL certificate for a given sight is replaced by your own self signed certificate. Inspection is only supported using an Internet Authority signed certificate.

Work on seamless OpenStack update/upgrade OpenStack separates the availability of workloads (Data Place) from the availability of the API services (Control Plane) OpenStack Code Major Upgrade Maintenance: Moving from one OpenStack release to another, e.g. from Kilo to Mitaka with data migration •  Infrequent, twice a year at most. New OpenStack code is put in place for all of the services that make up a cloud (e.g. Nova,

Neutron, Glance, Heat, Cinder, Ceilometer, Aodh, Swift, etc..). •  ~10 minutes for each service

OpenStack Code Minor Upgrade Maintenance: Upgrading OpenStack service, without data migration •  Typically only bug fixes. New OpenStack code is put in place and the services associated with that code are simply restarted.

•  ~5 seconds for each service

OpenStack Config Maintenance: Changing the configuration for one or more of the OpenStack services, needed to correct bugs in the service, or to tune the service due to consumer requests. •  Require the restart of a particular service. The disruption to the API control plane is isolated to the services receiving configuration

updates. •  ~ seconds for each service

© IBM Corporation 24

Lessons Learnt: Automate everything

OpenStack Installation: •  Leverage the open source Ursula Ansible and Rally Cloud infrastructure Automation framework •  Requires information about hardware, network environment and software repositories.

Bluemix Private Cloud Install Automation

Setup Storage

Setup OpenStack

Setup Network

Run validation

Setup Hardware

Ursula, Rally-OpenStack

OpenStack Discovery:

•  Leverage the open source Fog gem to discover OpenStack artifacts in an automated manner •  Require OpenStack credentials and discover OpenStack compute and network information.

Cloud Foundry on OpenStack deployment Automation

Discover VM Configuration Sizes

Discover Network Subnets

Discover Network Security Rules

Discover DHCP , DNS Gateway and floating IPs

Discover Security Credentials

Cloud Foundry Pre-req setup on OpenStack:

•  Leverage the open source Fog gem to setup Cloud Foundry requirements in an automated manner •  Setup according to best practices and guidelines – still giving users the flexibility to change if desired

Create Security Credentials

Create VM configs for Router, DEAs, Cloud Controller, Service Nodes

Create network Security Rules

Setup tenant quota

Cloud Foundry on OpenStack deployment Automation

Cloud Foundry Deployment Automation •  Automate stemcell modification •  Automate Cloud Foundry deployment manifest file genration using Ruby ERB •  Automate upload of Cloud Foundry core release, services and runtime frameworks, followed by Cloud

Foundry deployment

Stemcell Creation and Upload

Generate BOSH and Cloud Foundry Manifest

Upload Cloud Foundry core, Services and runtime

Deploy Cloud Foundry

Deploy bosh director

RUBYBOSH

Cloud Foundry on OpenStack deployment Automation

© IBM Corporation 29

As a Service: As a provider, look into offering Cloud Foundry and OpenStack as a Service

Why as a Service? Cloud Foundry:

–  New release every 2-3 weeks

–  Bluemix PaaS is a combination of CF and 150+ Services

–  Older versions will lead to huge version mismatches, and lead to version sprawl

–  Keeps Public/Dedicated and Local Bluemix in sync

OpenStack:

–  Twice annual releases that touch the entire code base.

–  Upgrading sequentially is important: stay up to date!

–  OpenStack’s complexity requires expertise in many operational areas

–  Focus on higher business value. Work with OpenStack, not on OpenSTack

Private Cloud Hardware

Bluemix Private Cloud: IaaS Relay: Box Panel

Box Panel

Site Controller (Software) B

luem

ix

Priv

ate

Clo

ud

Ope

nSta

ck

Box Panel Formations

Central Authentication Customer Relationship Management Service Catalog and Metering Billing and Invoicing

Obj

ect

Sto

rage

Blo

ck

Sto

rage

Cor

e N

etw

orki

ng

Inventory Management Network Management Reporting and Analytics Support Ticking, Chat and Email

IaaS Relay: Box Panel

IaaS Relay: Site Controller

Box Panel

Site Controller

Customer Cloud

Resides on-premises adjacent to customer clouds, providing real-time administrative control of cloud environments.

–  Network Automation

–  Power Distribution Unit Automation

–  System and Network: •  Monitoring

•  Telemetry

•  Logging

–  Secure Remote Control and Access

–  Bare Metal Provisioning

–  Package and Container Repo

SDN router

The Internet Customer Network

IBM Urban Code Deploy

Softlayer Server

Bluemix Platform

Stemcells Releases Manifests

BOSH CLI

Automated

Management Processes

(Deploy, Upgrade etc.)

IBM Urban Code Deploy

Relay

Customer Hardware & Infrastructure

Bluemix Core Services •  Monitoring & Logging

•  Cache

•  Cloudant (Data Store)

•  Qradar EPS

•  IEM Relay

Configuration Store

(per customer)

Bluemix Code &

Automation Repository

Opensource Code

IBM Test & Staging Validation

IBM Production Deployment &Validation

Bluemix Local

Inception VM

UCD Agent

• Secure connection • Connection originated from customer premise

• Restricted access (agent-only)

Cloud Foundry

ACE UI

Enterprise ITSM

Customer Services

Customer Premises

Premises

On-premise data store for logs, monitoring data etc.

Bluemix Ops Console

IBM Cloud Security Services

Qradar Console IEM Server

Bluemix Ops Directory Server

Privileged ID Governance

IPSec Tunnel

DataPower

• Customer’s Service traffic

• Syndicated & 3rd party service traffic

• App staging artifacts • Inbound & outbound user to app traffic

•  LDAP

•  Enterprise services

•  Other SaaS services •  vCenter

Network Isolation Customer Services

Bluemix Private Cloud PaaS Relay IBM Premises

PaaS Relay: Operations Console

PaaS Relay: Operations Console

VPN Tunnel

Inception VM

Stemcells

Releases

Manifests

BOSH CLI

DataPower

ACEUI

Metering

AdminUI

NATS

BMDB

Backup

Loginserver

UAA CC

Blobstore

HMCCDB

Loggregator

Gorouter

DEAs

UAADB

Logging

UCD Agent Ops Center Agent

Enterprise ITSM

NetworkIsolation•  LDAP

•  Enterprise services •  Other SaaS services

Customer Services

VPN Tunnel

Bluemix Private Cloud: PaaS and IaaS As a Service!

Bluemix Relay

Bluemix Private Cloud Relay

Site Controller

Repo / Deploy Server Monitoring Server

(Sensu) Logging (ELK)

Bastion (Access)

Cloud Foundry on OpenStack – A Great Fit!

•  100% Open PaaS and IaaS solutions – No vendor lock-ins •  Strong and growing community of contributors and sponsors on both sides

•  Power of Open Source community can be leveraged to automate the deployment and

lifecycle management of Cloud Foundry on OpenStack

•  OpenStack meets Cloud Foundry integration requirements, and is totally configurable and adaptable to handle the scale of a PaaS solution like Cloud Foundry

•  Bottom Line: It’s a match made in Heaven !