Optimizing Cloud Foundry and OpenStack for large scale deployments

Click here to load reader

  • date post

  • Category


  • view

  • download


Embed Size (px)



Transcript of Optimizing Cloud Foundry and OpenStack for large scale deployments

  • 1 OpenStack Summit May 12-16, 2014 Atlanta, Georgia Optimizing OpenStack for large scale Cloud Foundry deployments Animesh Singh, Daniel Krook, Jason Anderson, Kalonji Bankole @animeshsingh @danielkrook @andersonljason @k_bankole
  • 2 2014 IBM Corporation OAuth OSLC Infrastructure as a Service Platform as a Service Software as a Service API economy Cloud operating environment Software- defined environment TOSCA http://www.ibm.com/developerworks/cloud/library/cl-open-architecture/ IBM cloud and open technologies At all tiers, IBM is committed to building its cloud on an open cloud architecture Products and services built on open source and open standards benefit IBM and customers
  • 3 2014 IBM Corporation Gold Sponsors IBM is #2 in contributions to integrated projects IBM is working to accelerate OpenStack Foundation success Because an open interoperable Cloud is critical for flexible cloud deployment and customer success IBM has 15 core contributors15 IBMers working on OpenStack from formation of the Foundation to Code Quality & New Function380 IBMers active in the projects 2 107 Mar 2013 May 2014859 Contributors 8,500 Individuals 2130 Contributors 16,100+ Individuals Exponential growth Platinum Sponsors OpenStack Participant Growth
  • 4 2014 IBM Corporation4 2014 IBM Corporation And embracing Cloud Foundry as an open source PaaS Continuing our history of strengthening and extending open source TOTAL CONTRIBUTORS LINES OF CODE PULL REQ / WK COMPANIES 1,274 Average per month 2013 average: 133 12mo average: 98 Lifetime average: 58 711k 50+ 43
  • 5 2014 IBM Corporation Cloud Foundry PaaS Cloud Foundry services registry and runtime management layer. Components are dynamically discoverable and loosely coupled, exposing health through HTTP endpoints so agents can collect state and act on it. Cloud Foundry architectural overview User Authentication and Authorization Router DEA Pool Service Gateway Apps Service Connector Health Manager Messaging Cloud Controller Build Packs Cloud Foundry BOSH Micro Clouds Private Clouds Public Clouds Service Nodes
  • 6 2014 IBM Corporation Integrate!
  • 7 2014 IBM Corporation PaaS BOSH Cloud Provider Interface (CPI) Initially written for VMware and Amazon Web Services OpenStack CPI OpenStack CPI written by PistonCloud Apache2 license PistonCloud - An enterprise OpenStack company, founded by few of OpenStack founders Cloud Foundry OpenStack integration OpenStack Cloud Servers Networking Storage Integration IaaS Hardware
  • 8 2014 IBM Corporation BOSH deployment process Deployment Manifest Release name/version # VMs, job params Stemcells to use Stemcell Base OS BOSH agent Release Name Jobs Software packages Config templates Scripts BOSH Deployed Environment Virtual Machine Configuration Software Packages Virtual Machine Configuration Software Packages Virtual Machine Configuration Software Packages Virtual Machine Configuration Software packages
  • 9 2014 IBM Corporation Sample manifest
  • 10 2014 IBM Corporation 10 Cloud Provider Interface for OpenStack OpenStack CPI is an implementation of the BOSH CPI. Leverages the Fog Ruby gem for OpenStack OpenStack CPI gem (Ruby package), the OpenStack Registry and Agents all reside on the Cloud Foundry Bosh repository https://github.com/cloudfoundry/bosh Stemcell Image (Base OS Image) and sample manifest files for OpenStack exist
  • 11 2014 IBM Corporation Requirements: Static / floating ips Persistent disks Outbound Internet connectivity Custom flavors Increased quota Security groups 11 Integration requirements for OpenStack
  • 12 2014 IBM Corporation Automate!
  • 13 2014 IBM Corporation OpenStack discovery: Extend the Fog OpenStack gem used by CPI to discover and OpenStack artifacts in an automated manner in the Cloud Foundry manifest Pass OpenStack credentials and discover OpenStack compute and network information. Deployment automation: Tip one Discover Flavors Discover Networks Discover VM Subnet Discover DHCP IP Discover Keypairs Discover Gateway IP Discover Security groups
  • 14 2014 IBM Corporation Deployment automation: Tip two OpenStack pre-req setup: Extend the Fog OpenStack gem used by CPI to setup Cloud Foundry OpenStack pre-reqs Setup tenant quota Keypairs Custom Flavors Security Groups setup Create Keypair Create Flavors Router, DEA, Cloud Controller, Service Nodes Create Security Groups Cf-private, cf-public Setup tenant quota
  • 15 2014 IBM Corporation Deployment automation: Tip three Cloud Foundry manifest generation: Hard to manually create and edit Cloud Foundry manifests which can extend up to 1000+ lines Automation around template creation an extension from the previous efforts of OpenStack Discovery and Cloud Foundry pre-req setup. Can create templates for both nova-network and neutron based environments Iterates through jobs and assigns static IPs to them if we are using an environment without PowerDNS.
  • 16 2014 IBM Corporation Deployment automation: Tip four Stemcell automation: Community stemcell image has empty fstab and this makes the root partition auto detection to fail, and consequently provisioning to fail. Multiple Solutions: 1. Inject file into stemcell Mount stemcell image using qemu-nbd Inject /etc/fstab into mounted os Create new image based on snapshot of modified os 2. Modify Openstack Configuration: Change libvirt_inject_partition parameter in nova.conf from -1 (autodetect) to 1 (use first partition) 3. Modify bosh Stemcell Builder script Script injects /etc/fstab into CentOS, but not Ubuntu (as seen here) 4. Install cloud-init in the image Best solution to make it work in conjunction with metadata service Stemcell Base OS BOSH Agent
  • 17 2014 IBM Corporation Scale!
  • 18 2014 IBM Corporation Sample CF sizing for 1,000 small applications ~60 virtual machines 20 high memory DEAs 11 CF fabric components 26 service broker and service nodes 160 CPU ~500 GB memory ~1.5 TB VM disk ~200 GB block storage
  • 19 2014 IBM Corporation Controller Node Our initial OpenStack cluster was built on powerful hardware but it was not ideal for Cloud Foundry Compute Nodes Compute Nodes Compute Node Nova Compute Nova Network Storage Node Cinder (volume) Glance (image) 1x 20x Dedicated Servers (bare metal) Nova API Nova Scheduler Keystone Horizon Database (MySQL) Messaging (Qpid) 1x 128 GB memory 6 cores 36 TB disk each
  • 20 2014 IBM Corporation Weaknesses in the OpenStack layout affected the stability of our Cloud Foundry configuration Single points of failure in the networking, messaging, and storage configuration made BOSH unhappy. Qpid on Folsom memory leaks made the head node a bottleneck. Lost messages affected VM and storage creation, leaving many resources in stuck states. Single Nova API process design was overloaded with BOSH traffic VLAN per tenant design routed much traffic through the single head node, impacting both the fabric and brokered services. Cinder and Glance storage and APIs were located on a single node. Bare metal allocation of resources was also inefficient Every OpenStack component ran as a process directly on the OS Made reacting to workload changes difficult, i.e., difficult to redistribute OpenStack components to react to Cloud Foundry demands
  • 21 2014 IBM Corporation Weve addressed the major issues OpenStack by distributing components more intelligently Moving some components to VMs, assigning controller, compute, data, storage roles to groups of components improves availability, flexibility, scalability and maintainability Compute Nodes Compute Nodes Compute Node Nova Compute Cloud Controller Data Node database (mysql) messaging (qpid) Cloud Controller Master Node nova-api nova-scheduler Keystone Horizon Cloud Controller Data Node database (MySql) messaging (Qpid) Cloud Controller Storage Node Cinder (volume) Glance (image) Nova Network Controller Node Nova API Nova Scheduler Keystone Horizon Storage Node Cinder (volume) Glance (image) 2x Data Node Database (MySQL) Messaging (RabbitMQ) 2x 15x Load Balancer Node HA Proxy Keepalive Load Balancer Node HA Proxy Keepalive Load Balancer Node HA Proxy KeepAlive 6x3x Virtual Machines (VMs) Dedicated Servers (bare metal)
  • 22 2014 IBM Corporation Matching the right hardware for OpenStack to host Cloud Foundry Learn more about these topics at sessions from our team Wednesday
  • 23 2014 IBM Corporation Optimize!
  • 24 2014 IBM Corporation Deplo