Cloud Foundry V2 | Intermediate Deep Dive
69
Cloud Foundry V2 Intermediate Deep Dive
-
Upload
kazuto-kusama -
Category
Technology
-
view
1.627 -
download
3
Transcript of Cloud Foundry V2 | Intermediate Deep Dive
Cloud Foundry V2 Intermediate Deep Dive
@jacopenKazuto KusamaNTT Communications
Developer for Cloudn PaaS
Last Time
http://www.slideshare.net/jacopen/cloud-foundry-33851040
Summary of Last time
• Cloud Controller• Provides API and control
each components• Router
• Dispatch access to API and Apps• DEA
• Runs user applications• Health Manager
• Monitor user application health status
Cloud Controller
Router
DEAHealthManage
r
Components are defined according to role
Summary of last time
Each components are connected via light weight messaging system called NATS. They are loosely coupled.
There is no big difference between V1 and V2 in overall
architecture of Cloud FoundryCloud Controller
Router
DEA
Health
Manager
NATS
So what’s the difference？
Agenda
3 points to understand Cloud Foundry V2
• What is Buildpack• How Buildpack and application relate each other to
run on Cloud Foundry• What is Warden to secure application.
Cloud Foundry
Buildpack1
Cloud FoundryHeroku
Buildpack1
Buildpack
A Buildpack provides arbitrary language and framework runtime support. Originally created by Heroku
In case of Heroku
$ heroku create myapp -b https://github.com/kr/heroku-buildpack-go.git
In this case, Golang can be used which Heroku does
not officially support.
In case of Cloud Foundry
$ cf push myapp -b https://github.com/dmikusa-pivotal/cf-php-build-pack.git
Buildpacks in CF
•Cloud Foundry Buildpack specification comply with Heroku specification.
•They should have basically compatibility between them.
• In reality, there are difference in the environment (OS, Linux container, network, etc). Some Heroku Buildpacks do not work very well.
Buildpack fundamental
Element of Buildpack
•detect•compile•release
1. detect
detect
•A script to detect condition to run Buildpack.•Runs this detect script against deployed application. OK if exit 0, NG if exit 1
heroku-buildpack-php detect
#!/usr/bin/env bash
if [[ -f "$1/composer.json" || -f "$1/index.php" ]]; then echo "PHP" && exit 0else exit 1fi
If there is composer.json or index.php, then this
Buildpack is used.
heroku-buildpack-ruby detect
#!/usr/bin/env ruby
require 'pathname'
if Pathname.new(ARGV.first).join("Gemfile").exist? puts "Ruby" exit 0else puts "no" exit 1end
If there is Gemfile, then this Buildpack is used.
detect
•Can write anything if it is executable. Heroku-buildpack-php is written in bash. heroku-buildpack-ruby is written in ruby.
•There are some examples that detect itself is written bash, and then calls python script (e.g. cf-php-build-pack)
•Return 0 or 1 anyway.
2. compile
compile
•Key script to setup language/framework runtime setup.
•As the name suggest, it downloads execution environment sources, and then compiles. However, because it takes time to do compile, many Buildpacks download pre-compiled binaries.
3. release
release
•A script to return metadata required for execution in yaml format.
#!/bin/sh
cat << EOF---addons: - heroku-postgresql:devdefault_process_types: web: bin/node server.jsEOF
#!/bin/sh
cat << EOF---addons: - heroku-postgresql:devdefault_process_types: web: bin/node server.jsEOF
addons: Describes required addons by the Buildpack. Currently this is not used in CF.
default_process_types: Describes application execution command. Currently CF only supports web framework.
Staging using Buildpack２
”Staging” in CF V2
•A work to build Droplet•Droplet is a executable bundle of application.
• In CF V2, staging means- User deploy application.- Process the application by Buildpack description- Then create Droplet
(staging.start)The chart used in the last time
Cloud Controller
Router
DEAHealth
Manager
cf push
Gemfilelib/bin/config.ruapp.rb
GemfileGemfile.lockvendor/lib/bin/config.ruapp.rb
1.Via “cf push“, user upload application to Cloud Controller
2.Cloud Controller send “staging.start” message to DEA to request Staging activity.
3.DEA works on Staging activity, build Droplet
4.Once Droplet is built, it returns back to Cloud Controller
Deep dive this proces
{ "app_id": "65bf0610-fb24-4756-a49a-b64edbe456ed", "task_id": "3b638a01c39b4187a64c76e3ae89a5dd", "properties": { "services": [ ], "resources": { "memory": 256, "disk": 1024, "fds": 16384 }, "environment": [ ], "meta": { "console": false } }, "download_uri": "http://upload-user:[email protected]:9022/staging/apps/65bf0610-fb24-4756-a49a-b64edbe456ed", "upload_uri": "http://upload-user:[email protected]:9022/staging/droplets/65bf0610-fb24-4756-a49a-b64edbe456ed/upload", "buildpack_cache_download_uri": null, "buildpack_cache_upload_uri": "http://upload-user:[email protected]:9022/staging/buildpack_cache/65bf0610-fb24-4756-a49a-b64edbe456ed/upload", "start_message": { "droplet": "65bf0610-fb24-4756-a49a-b64edbe456ed", "name": "dora", "uris": [ "dora.107.22.72.200.xip.io" ], "prod": false, "sha1": null, "executableFile": "deprecated", "executableUri": null, "version": "32395f10-ce09-40c1-99c2-e0f317d66e51", "services": [ ], "limits": { "mem": 256, "disk": 1024, "fds": 16384 }, "cc_partition": "default", "env": [ ], "console": false, "debug": null, "start_command": null, "health_check_timeout": null, "vcap_application": { "limits": { "mem": 256, "disk": 1024, "fds": 16384 }, "application_version": "32395f10-ce09-40c1-99c2-e0f317d66e51", "application_name": "dora", "application_uris": [ "dora.107.22.72.200.xip.io" ], "version": "32395f10-ce09-40c1-99c2-e0f317d66e51", "name": "dora", "space_name": "jacopen-space", "space_id": "33ab8f3b-1a1f-40a7-933d-d2f55a1c80c4", "uris": [ "dora.107.22.72.200.xip.io" ], "users": null }, "index": 0 }, "admin_buildpacks": [ { "key": "9130b02f-9938-4994-9d95-3c864eb1f7b2_f05a183eea56c23f90d8fd616694d8e6a1627ddf", "url": "http://upload-user:[email protected]:9022/v2/buildpacks/9130b02f-9938-4994-9d95-3c864eb1f7b2/download" }, { "key": "c23f44c8-16e4-44bf-a4ed-b228bd05c9a9_ea13962fc6930f2b0d82be4d87ba8222cd3fab6a", "url": "http://upload-user:[email protected]:9022/v2/buildpacks/c23f44c8-16e4-44bf-a4ed-b228bd05c9a9/download" }, { "key": "42d73ff8-20a2-4ff3-bb52-c36444186c84_e6c7c383baf52d8e9cfbee8042cd62ac1dc5798a", "url": "http://upload-user:[email protected]:9022/v2/buildpacks/42d73ff8-20a2-4ff3-bb52-c36444186c84/download" } ]}
staging.start example
Expand
{ "app_id": "65bf0610-fb24-4756-a49a-b64edbe456ed", "task_id": "3b638a01c39b4187a64c76e3ae89a5dd", "properties": { "services": [ ], "resources": { "memory": 256, "disk": 1024, "fds": 16384 }, "environment": [ ], "meta": { "console": false } },
Application basic information
"download_uri": "http://upload-user:[email protected]:9022/staging/apps/65bf0610-fb24-4756-a49a-b64edbe456ed", "upload_uri": "http://upload-user:[email protected]:9022/staging/droplets/65bf0610-fb24-4756-a49a-b64edbe456ed/upload", "buildpack_cache_download_uri": null, "buildpack_cache_upload_uri": "http://upload-user:[email protected]:9022/staging/buildpack_cache/65bf0610-fb24-4756-a49a-b64edbe456ed/upload",
Download URL of user application
Upload location of completed Droplet
Download URL of Buildpack cache
Upload location of Buildpack cache
"start_message": { "droplet": "65bf0610-fb24-4756-a49a-b64edbe456ed", "name": "dora", "uris": [ "dora.107.22.72.200.xip.io" ], "prod": false, "sha1": null, "executableFile": "deprecated", "executableUri": null, "version": "32395f10-ce09-40c1-99c2-e0f317d66e51", "services": [ ], "limits": { "mem": 256, "disk": 1024, "fds": 16384 }, "cc_partition": "default", "env": [ ], "console": false, "debug": null, "start_command": null, "health_check_timeout": null, "vcap_application": { "limits": { "mem": 256, "disk": 1024, "fds": 16384 }, "application_version": "32395f10-ce09-40c1-99c2-e0f317d66e51", "application_name": "dora", "application_uris": [ "dora.107.22.72.200.xip.io" ], "version": "32395f10-ce09-40c1-99c2-e0f317d66e51", "name": "dora", "space_name": "jacopen-space", "space_id": "33ab8f3b-1a1f-40a7-933d-d2f55a1c80c4", "uris": [ "dora.107.22.72.200.xip.io" ], "users": null }, "index": 0 },
How to run the application, etc
"admin_buildpacks": [ { "key": "9130b02f-9938-4994-9d95-3c864eb1f7b2_f05a183eea56c23f90d8fd616694d8e6a1627ddf", "url": "http://upload-user:[email protected]:9022/v2/buildpacks/9130b02f-9938-4994-9d95-3c864eb1f7b2/download" }, { "key": "c23f44c8-16e4-44bf-a4ed-b228bd05c9a9_ea13962fc6930f2b0d82be4d87ba8222cd3fab6a", "url": "http://upload-user:[email protected]:9022/v2/buildpacks/c23f44c8-16e4-44bf-a4ed-b228bd05c9a9/download" }, { "key": "42d73ff8-20a2-4ff3-bb52-c36444186c84_e6c7c383baf52d8e9cfbee8042cd62ac1dc5798a", "url": "http://upload-user:[email protected]:9022/v2/buildpacks/42d73ff8-20a2-4ff3-bb52-c36444186c84/download" } ]}
A list of admin_buildpack
Cloud Controller
DEA
Download the application here, and build the app.
Then when finised upload to this location. (Note)
If you succeeded Staging, then Run the application too !
This is the Buildpack for the Staging
Note: It is not desirable to send/receive large binaries using NATS, HTTP is used for this case.
So staging.start is
admin buildpack
•A Buildpack directly installed in Cloud Foundry
•When user did not specify Buildpack, then CF selects appropriate Buildpack from admin buildpack
•Administrator can add using create-buildpack
•As a result, CF service provider can add “Supported languages” to the platform.
DEA
staging.start is received by DEA
DEA
Execute admin buildpack detect based on
priorities
Gemfilelib/bin/config.ruapp.rb
Ruby
Java
node
It must be Ruby because it has
Gemfile
DEA
Run Buildpack compile which matched
Gemfilelib/bin/config.ruapp.rb
Ruby
• Download Ruby binaries
• Setup binaries
• Execute “bundle install”
• Replace database.yml (in case of Rails)
• Etv etc
DEA
Execute release, and produce nessesary information
Gemfilelib/bin/config.ruapp.rb
---buildpack_path: /var/vcap/data/dea_next/admin_buildpacks/c23f44c8-16e4-44bf-a4ed-b228bd05c9a9_ea13962fc6930f2b0d82be4d87ba8222cd3fab6adetected_buildpack: Ruby/Rackstart_command: bundle exec rackup config.ru -p $PORT
staging_info.yml
DEA
Droplet contents├── app│ ├── bin│ │ ├── erb -> ../vendor/ruby-1.9.3/bin/erb│ │ ├── gem -> ../vendor/ruby-1.9.3/bin/gem│ │ ├── irb -> ../vendor/ruby-1.9.3/bin/irb│ │ ├── rake -> ../vendor/ruby-1.9.3/bin/rake│ │ ├── rdoc -> ../vendor/ruby-1.9.3/bin/rdoc│ │ ├── ri -> ../vendor/ruby-1.9.3/bin/ri│ │ ├── ruby -> ../vendor/ruby-1.9.3/bin/ruby│ │ └── testrb -> ../vendor/ruby-1.9.3/bin/testrb│ ├── config.ru│ ├── dora.rb│ ├── Gemfile│ ├── Gemfile.lock│ ├── get_instance_cookie_jars.sh│ ├── instances.rb│ ├── loop.sh│ ├── README.md│ ├── routes.txt│ ├── spec│ │ ├── instances_spec.rb│ │ ├── spec_helper.rb│ │ └── stress_testers_spec.rb│ ├── stress│ ├── stress_testers.rb│ └── vendor│ ├── bundle│ ├── cache│ ├── heroku│ └── ruby-1.9.3├── logs│ └── staging_task.log├── staging_info.yml└── tmp
Ruby executables created byBuildpack
Uploaded applica-tion
Files installed by bundle install
Start command description
custom Buildpack
•User can specify Buildpack at the timing of deploy. This is called custom Buildpack.
•When custom Buildpack is specified, detect step is skipped, compile step is executed.
$ cf push myapp -b https://github.com/dmikusa-pivotal/cf-php-build-pack.git
What happens when Buildpack is upgraded
Buildpack update
• In case of Ruby Buildpack, the latest patch level version is deployed. For example, in case of Ruby 2.0.0, the latest is p481 (at the timing of wrinting).
•What happens when latest patch level has been published after the deployment ?
DEMO
Summary of Demo• Just stop and start does not update Buildpack.
• This is because last build Droplet is used.
• Re staging (e.g. cf push) will use new Buildpack during deploy.
• Recent CF release C171 has restaging API.
• But cf does not have this feature.
Container to run application.
Warden３
Warden
• wˈɔɚdn• A container to isolate user application.• User application runs inside Warden container. When
the application stops, then container is get destroyed.• Currently, in addition to run the application, staging
process itself and also used to run component in Bosh-lite environment.
Why container technology- The problem with UNIX user
• Vulnerability- elevated privilege
• Affected by another application- CPU, Memory, IO, etc
• Information propagation- PID, CPU, Memory, etc- Disgusting…
DEA Host (OS)
App App App
DEA Host (OS)
AppApp
App
DEA Host (OS)
App AppApp
http://www.slideshare.net/i_yudai/warden
Pstree on DEA VM
DEA(ng) = dea_next + warden
DEA
dea_next(Ruby)
warden(Ruby)
DEA(ng) = dea_next + warden
DEA
dea_next(Ruby)
warden server(Ruby)
containerwshd (C)
Warden Protocol
Warden Client
Start Request
Warden
• dea_next uses warden client to invoke warden server API.
• Communication between warden is determined by warden protocol.
• Warden server is written in Ruby. However the container which warden server runs is written in C.
Technologies around Warden
• namespaces (space isolation)• NET• MNT• PID• UTS• IPC
• cgroups(CPU, memory restriction)• Copy on write via aufs(10.04), overlayfs(12.04)
Network
• Virtual NIC is allocated to container.• NAT using iptables against virtual NIC.
Almost same
• LXC• Docker
Frequently asked Questions
Why Warden (vs LXC)
• In fact, initial CF used LXC. • LXC is based on Linux. In the future, CF wants to
cover the platforms other than Linux, framework for multiple platform was desirable.
• LXC is fully overloaded against CF requirements. It only required under 1000 lines of codes, and required minimal features. In stead, simplicity and more transparency were demanded.
https://github.com/cloudfoundry/warden/tree/master/warden
Why Warden(vs Docker)
• Docker only runs a single process, it does not conform CF architecture.
• Can not control limit after the container creation.• Can not control disk and bandwidth limits.
https://docs.google.com/document/d/1DDBJlLJ7rrsM1J54MBldgQhrJdPS_xpc9zPdtuqHCTI/edit
Infrastructure as Code
HerokuBuildpack ⇒ Runs on LXC
Cloud FoundryBuildpack Runs on Warden⇒
DockerDockerfile Runs on Docker⇒
Personal opinion
• The main purpose is ultimately the same in Docker, Heroku and CF. There is no meaning to replace Warden to Docker.
• Personally, it is easier to write Buildpack than Dockerfile.
• Having said that, I want to try deploy Docker image on my local hand.
Other info.
• CloudCredo enabled to deploy using Dockerfile, called Decker.
• ActiveState introduced Docker in Stackato 3.0 which is based on Cloud Foundry. Previously, they used LCX.
• DEA next architecture Diego may have some option to support Docker in someway. Basically it uses Go version of Warden (Garden).
Questions ?
End Note.
This material was translated by @ibmamnt
From original chart by @jacopen
http://www.slideshare.net/jacopen/c-fv2
