Post on 14-Jan-2017
Copyright © NTT Communications Corporation. All rights reserved.
Automated Deployment & Benchmarkingwith Chef, Cobbler and Rally for OpenStack
NTT Communications Corporation
Yuki NishiwakiMahito OguraTakeaki Matsumoto
Copyright © NTT Communications Corporation. All rights reserved.
About Us
1
Mahito Ogura m.ogura@ntt.com
Technology DevelopmentDevOps Engineer
R&D for IaaS, NoSQL, DB as a Service,Hadoop as a service
Contributing to Devstack and Rally
Yuki Nishiwaki yuki.nishiwaki@ntt.com
Technology DevelopmentSoftware Engineer
R&D for IaaS
Contributing to Devstack and Rally-‐‑‒ 11 patches committed in Liberty
Copyright © NTT Communications Corporation. All rights reserved.
Agenda
2
1. Company Overview
2. Automated Deployment & Benchmarking– Deployment– Testing– Benchmarking– Evaluation
3. Use Cases
4. Conclusion
Copyright © NTT Communications Corporation. All rights reserved.
About NTT Communications
3
OpenStack in Production• Cloudn VPC type ClosedNW• Enterprise Cloud(coming soon)
Contribution to OpenStack Community• OpenStack Foundation Corporate Sponsor• Japan OpenStack user group• Bug Report / Bug Fix / Blueprint
Copyright © NTT Communications Corporation. All rights reserved.
OpenStack in Production
4
Launched our first OpenStack product in October 2013
Copyright © NTT Communications Corporation. All rights reserved.5
OpenStack deployment is not easy thing to do• So many components/modules have to be synced and integrated
• Environment needs to be updated for bug-‐‑‒fix and enhancement
• There is no magic script to make environment up and running each time we deploy
OpenStack Deployment
Copyright © NTT Communications Corporation. All rights reserved.
Our Challenge
6
How can we continuously deploy OpenStack while consistently keeping service performance?
Copyright © NTT Communications Corporation. All rights reserved.
2. Automated Deployment & Benchmarking
7
Copyright © NTT Communications Corporation. All rights reserved.
Improvement cycle
8
Functional Test Benchmark
Output Result
Change structure
Deploy OpenStack
Evaluate Result
Copyright © NTT Communications Corporation. All rights reserved.
Benchmark
Our Implementation (1/2)
9
Functional Test
Output Result
Change structure
Automate Automate
Hand work
Deploy OpenStackChef RallyTempest
Cobbler
Evaluate Result Hand workFluentd Grafana InfluxdbGithub
EnterpriseRally
Copyright © NTT Communications Corporation. All rights reserved.
Benchmark
Our Implementation (2/2)
10
Functional Test
Output Result
Change structure
Automate Automate
Hand work
Deploy OpenStack
Hand work
Chef RallyTempestCobbler
Evaluate Result Fluentd Grafana InfluxdbGithub
EnterpriseRally
Copyright © NTT Communications Corporation. All rights reserved.
Deployment
11
Functional Test
Output Result
Change structure
Automate Automate
Hand work
Deploy OpenStack
Hand workEvaluate Result
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
Automated deployment is divided into four steps• Orchestration• Configuration• Bootstrapping• Networking
Deployment Steps Deployment
12
Copyright © NTT Communications Corporation. All rights reserved.
Tools for Automation
Orchestration
Configuration
Bootstrapping
Networking
Setting for clusters
Setting for just one server
Installing OS into physical servers
Setting up network switch
13
Chef
Expect
Cobbler
Deployment
Chef
Copyright © NTT Communications Corporation. All rights reserved.
Infrastructure as a Code
Environmental information is maintained in text format• Definition of network
• System architecture of OpenStackü Which role does the server have
• Setting, Configuration of Server ü Chef Repository
Version control is managed by Git
Deployment
14
Copyright © NTT Communications Corporation. All rights reserved.
Deployment flow
Git clone Update Chef Server
Settingswitch
OS installApply Chef(Configuration
role)
Apply Chef(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping Configuration Orchestration
Deployment
15
Copyright © NTT Communications Corporation. All rights reserved.
Deployment flow
Git clone Update Chef Server
Settingswitch
OS installApply Chef(Configuration
role)
Apply Chef(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping Configuration Orchestration
Deployment
16
Copyright © NTT Communications Corporation. All rights reserved.
Deployment flow
Git clone Update Chef Server
Setting switch
OS installApply Chef(Configuration
role)
Apply Chef(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping Configuration Orchestration
Deployment
17
Copyright © NTT Communications Corporation. All rights reserved.
Automate Networking
• Auto VLAN setup• Preconditions
ü Define server port nameü Set switch port description with naming rule
“<server_̲name>_̲<server_̲port_̲name>”
eth0 eth1 eth2 eth3 eth4 eth5
eth6 eth7
Naming rule for physical port-‐‑‒ server_̲1: -‐‑‒ eth0: -‐‑‒ vlan10: -‐‑‒ native -‐‑‒ vlan100 -‐‑‒ eth1: -‐‑‒ vlan10: -‐‑‒ native
Config.yaml
When I got servers, I decide it
Networking Deployment
18
Copyright © NTT Communications Corporation. All rights reserved.
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
Automate Networking flow (1/4)
19
Service
ChefServer
1
config.yaml (image)
-‐‑‒ server_̲1 -‐‑‒ eth0 -‐‑‒ vlan10 -‐‑‒ native -‐‑‒ vlan100-‐‑‒ eth1 -‐‑‒ vlan10
Networking Deployment
1. Picks up configuration including VLAN info
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Automate Networking flow (2/4)
20
Service
ChefServer
Networking Deployment
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
2. Gets port description from network switch
2
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
Automate Networking flow (3/4)
21
Service
ChefServer
Server1 eth0(switch1 port 1/15) => VLAN100, 200Server1 eth1(switch2 port 1/15) => VLAN100, 200
Networking Deployment
3. Generates switch configuration based on network layout and target server settings
3
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Automate Networking flow (4/4)
22
Service
ChefServer
Networking Deployment
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
4. Executes the commands which Jenkins generated
4
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Deployment flow
Git clone Update Chef Server
Setting switch
OS installApply Chef(Configuration
role)
Apply Chef(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping Configuration Orchestration
Deployment
23
Copyright © NTT Communications Corporation. All rights reserved.
Bootstrapping flow (1/3)
24
Service
ChefServer
Deployment Bootstrap
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
1. Run “chef-‐‑‒client” via ssh2. Apply chef
ü Install cobbler and configureü Register machines
ü hostnameü IP address
2
1
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Bootstrapping flow (2/3)
25
Service
ChefServer
Deployment Bootstrap
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
3. IPMI power reset4. Start auto install procedure (PXE)
3
4
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Bootstrapping flow (3/3)
26
Service
ChefServer
Ubuntu
Deployment Bootstrap
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
5. Check ssh connection periodically
5
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Deployment flow
Git clone Update Chef Server
Setting switch
OS installApply Chef(Configuration
role)
Apply Chef(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping Configuration Orchestration
Deployment
27
Copyright © NTT Communications Corporation. All rights reserved.
Configuration, Orchestration flow(1/3)
28
Internet
ChefServer
Ubuntu
Deployment
Rally
Tempest
influxdbCobbler
Configura2on
Orchestra2on
1. Git clone the definition file of the system architecture of OpenStack
{ “<environment>”: { “servers”: { “server1”: “backend”, “server2”: “controller” }, “dependencies”: { “backend”: [], “controller”: [{“backend”:1}] } }}
definition.json
1
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Configuration, Orchestration flow(2/3)
29
Internet
ChefServer
Ubuntu
Deployment
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
Configura2on
Orchestra2on
3. Apply chefü backend configuration roleü backend orchestration role…
3
2
2. Run “chef-‐‑‒client” via ssh in order of the definition file
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Configuration, Orchestration flow(3/3)
30
Internet
ChefServer
OpenStack
Deployment
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
Configura2on
Orchestra2on
4
Github Enterprise
4. Remove orchestration roleü divided role into configuration & orchestrationü doesnʼ’t have idempotence.
Copyright © NTT Communications Corporation. All rights reserved.
Deployment is ended
Git clone Update Chef Server
Setting switch
OS installApply Chef(Configuration
role)
Apply Chef(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping Configuration Orchestration
Deployment
31
Copyright © NTT Communications Corporation. All rights reserved.
Review the Automate deployment
32
Pros• Anyone can create the same environment anytime
Cons• Any single tool does not satisfy auto deployment
ü We need to use multiple tools in combinationü We need wide knowledge
Deployment
Copyright © NTT Communications Corporation. All rights reserved.
Functional Test
34
Functional Test
※ Omit detail process here
Output Result
Change structure
Automate Automate
Hand work
Deploy OpenStack
Hand workEvaluate Result
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
Functional Test
35
Tool for test• We use Tempest
What is Tempest?• Tempest is Official Integration Test Suite• Tempest can test any OpenStack, all in one or 1000 nodes
Functional Test
Copyright © NTT Communications Corporation. All rights reserved.
Functional Test
36
Tempest is used for upstream developing
Community Workflow by http://docs.openstack.org/
Functional Test
Copyright © NTT Communications Corporation. All rights reserved.
What types of test we use
37
We use• API Tests
ü Unit tests that ensure API works properly• Scenario Tests
ü Series of actions like actual users do
Functional Test
Copyright © NTT Communications Corporation. All rights reserved.
How to use
38
We can execute it in only two steps 1. Edit tempest.conf2. Execute ./run_̲tempest
$ ./run_̲tempest………Ran 1124 tests in 1185.267sFAILED (failures=4)
Functional Test
Copyright © NTT Communications Corporation. All rights reserved.
Test flow (1/4)
39
Functional Test
Service
ChefServer
OpenStack
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
1. Create a tempest container from the image
1
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Test flow (2/4)
40
Functional Test
Service
ChefServer
OpenStack
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
2. Run “chef-‐‑‒client” via ssh3. Apply chef recipe, To set up tempest.conf
3
2
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Test flow (3/4)
41
Functional Test
Service
ChefServer
OpenStack
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
4. Run tempest against OpenStack
4
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Test flow (4/4)
42
Functional Test
Service
ChefServer
OpenStack
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
5. Get the results and delete the container which I created right now
5
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Review the Test
43
Pros• Generate configuration file automatically based on auto deployed OpenStack
• Execute functional tests simultaneously with different customized Tempest
Cons• Doesnʼ’t cover system architecture
ü We proposed 11 patches for fixing tempest• Doesnʼ’t control test cases based on API level
Functional Test
Copyright © NTT Communications Corporation. All rights reserved.
Benchmark
45
Functional Test
※ Omit detail process here
Output Result
Change structure
Automate Automate
Hand work
Deploy OpenStack
Hand workEvaluate Result
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
Benchmark
46
Tool for benchmark• Rally ( is one of the OpenStack projects )
What is Rally ?• Released v0.1.1 in 6 October 2015• Benchmarking tool
ü Not only Benchmark but support functions of deploy and verification
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
47
1. Create deployment
rally@3eba20593250:~∼$ cat config.json
{ "admin": { "password": “p@ssw@rd", "tenant_̲name": ”admin", "username": ”admin" }, "auth_̲url": "http://lb:5000/v2.0", "type": "ExistingCloud"}
rally@3eba20593250:~∼$rally deployment create -‐‑‒-‐‑‒filename config.json -‐‑‒-‐‑‒name existing
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
48
2. Check credentialrally@3eba20593250:~∼$ rally deployment checkkeystone endpoints are valid and following services are available:+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+| services | type | status |+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+| cinderv2 | volumev2 | Available || glance | image | Available || keystone | identity | Available || neutron | network | Available || nova | compute | Available |+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
49
3. Create task filerally@3eba20593250:~∼$ cat benchmarks/nova/boot-‐‑‒and-‐‑‒delete.json{% set flavor_̲name = flavor_̲name or "m1.tiny" %}{ "NovaServers.boot_̲and_̲delete_̲server": [ { "args": { "flavor": { "name": "{{flavor_̲name}}" }, "image": { "name": "^cirros.*$" } } } ]}
Task file is expressed in json or yaml and We can use statement of Jinja2.
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
50
4. Start benchmarkrally@3eba20593250:~∼$ rally task start benchmarks/nova/boot-‐‑‒and-‐‑‒delete.json +-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+| Response Times (sec) |+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+| action | min | median| 90%ile| 95%ile | max | avg | success | count |+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+| nova.boot_̲server | 4.614 | 4.768 | 4.972 | 5.417 | 5.861 | 4.852 | 100.0% | 10 || nova.delete_̲server | 2.486 | 2.576 | 2.709 | 2.732 | 2.754 | 2.603 | 100.0% | 10 || total | 7.148 | 7.369 | 7.658 | 8.018 | 8.377 | 7.455 | 100.0% | 10 |+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+Load duration: 37.9388210773Full duration: 51.1114509106…
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
51
5. Make result “Human Readable” (Create HTML Report)rally@3eba20593250:~∼$ rally task report -‐‑‒-‐‑‒tasks 5e8815a9-‐‑‒10eb-‐‑‒4d67-‐‑‒86b4-‐‑‒6027f4adc634 1f71cb68-‐‑‒9c1e-‐‑‒49e3-‐‑‒8af5-‐‑‒6d21dc34c455 -‐‑‒-‐‑‒out result.html -‐‑‒-‐‑‒html
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
52
※1 There is blueprinthttps://blueprints.launchpad.net/rally/+spec/cli-‐‑‒plot2html-‐‑‒compare-‐‑‒tasks
We can see each detailed result of the benchmark. But we canʼ’t compare each result in HTML yet※1
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
How I use Rally
53
Benchmark
BenchmarkCompareto the past(Summary)
Nothingto do
CheckDetail
No
Detect unusual
Yes
Rally task report
Rally task start
Copyright © NTT Communications Corporation. All rights reserved.
How I use Rally
54
Benchmark
BenchmarkCompareto the past(Summary)
Nothingto do
CheckDetail
No
Detect unusual
Yes
Rally task report
BenchmarkDashboard
We created it
Rally task start
Copyright © NTT Communications Corporation. All rights reserved.
Benchmark Dashboard
55
Benchmark Dashboard
• Provides two functionsü Compare summarized result with past resultü Provide link to a html report created by “rally task report”
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
Benchmark flow (1/4)
57
Benchmark
Service
ChefServer
OpenStack
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
1. Create a container from the image which I prepared
1
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Benchmark flow (2/4)
58
Benchmark
Service
ChefServer
OpenStack
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
2. Register existing OpenStack3. Run each benchmark
2
3
Github Enterprise
Copyright © NTT Communications Corporation. All rights reserved.
Internet
Github Enterprise
Benchmark flow (3/4)
59
Benchmark
ChefServer
OpenStack
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
4. Upload report.html to Benchmark dashboard5. Post summarized results to influxdb
4
5
Copyright © NTT Communications Corporation. All rights reserved.
Service
Github Enterprise
Benchmark flow (4/4)
60
ChefServer
OpenStack
Rally
Tempest
influxdb
BenchmarkDashboard
Cobbler
Developer
view result
We can view the result at the dashboard
Copyright © NTT Communications Corporation. All rights reserved.
Review the Benchmark
61
Benchmark
Pros• We can compare the result with the past• The process of Benchmark is fully automated
Cons• We donʼ’t support many benchmark scenarios• Best practice for benchmark isnʼ’t fixed yet
Copyright © NTT Communications Corporation. All rights reserved.
Deploy and Test and Benchmark cycle
63
Functional Test
Output Result
Change structure
Automate Automate
Hand work
Deploy OpenStack
Hand workEvaluate Result
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
When benchmark score is bad
64
• Need to find bottlenecks or issuesü Too many host resourcesü Too many logsü Too difficult to find bottlenecks or issues ...
• Develop analysis systemü To enable data collectingü To enable monitoringü To enable alert notify
Copyright © NTT Communications Corporation. All rights reserved.
Architecture to evaluate the result
65
OpenStack logsHost resource data
Storage
Host resources Log information
and more …Search log data
notify
store
Get WARNING/ERROR notifications
Copyright © NTT Communications Corporation. All rights reserved.
Improvement cycle
66
Functional Test
Output Result
Change structure
Automate Automate
Hand work
Deploy OpenStack
Hand workEvaluate Result
Benchmark
Copyright © NTT Communications Corporation. All rights reserved.
What scenario I use
68
Benchmark Scenario• VMTask scenario ( which is contained in rally at default )
ü Prepare the task files which are different in concurrency(30, 50)
nova boot ssh loginrun
command via ssh
nova delete
Use case
Copyright © NTT Communications Corporation. All rights reserved.
Why I use
69
Why I use VMTask scenario• The scenario is the most common use case • We need to know which components will be affected by increasing the number of users
Use case
Copyright © NTT Communications Corporation. All rights reserved.
What happened
70
What happened when I execute the benchmark• Actually, at first success rate isnʼ’t 100%
Use case
Copyright © NTT Communications Corporation. All rights reserved.
Analyze cause of failure
71
Check the detailed result of the benchmark
it Doesnʼ’t have IP reachability.
Use case
Copyright © NTT Communications Corporation. All rights reserved.
Analyze cause of failure
72
Check the neutron log file$ less /var/log/neutron/neutron-‐‑‒server.log…TimeoutError: QueuePool limit of size 5 overflow 20 reached, connection timed out, timeout 10…
Overflow connection pool.Need to review the configuration about database.
Use case
Copyright © NTT Communications Corporation. All rights reserved.
Change configuration
73
Change [database] in neutron.conf-‐‑‒ max_̲retries-‐‑‒ retry_̲interval-‐‑‒ min_̲pool_̲size-‐‑‒ max_̲pool_̲size-‐‑‒ ide_̲timeout-‐‑‒ max_̲overflow
Use case
Copyright © NTT Communications Corporation. All rights reserved.
Check again
74
$ git checkout -‐‑‒b change_̲db_̲setting_̲in_̲neutron_̲conf$ git commit -‐‑‒am “foobar”$ git push origin change_̲db_̲setting_̲in_̲neutron_̲conf
Request Jenkins via Slack
Use case
Copyright © NTT Communications Corporation. All rights reserved.
Check again
75
This change improves the success rates
Use case
Copyright © NTT Communications Corporation. All rights reserved.
conclusion
77
• Default configuration doesnʼ’t cover all use cases
• We need to investigate the best configuration
• Best configuration requires us to follow the cycleü deployment, functional test, benchmark, evaluation
• Any single tool cannot automate whole processü multiple Open Source Software can automate whole process
Copyright © NTT Communications Corporation. All rights reserved.
License
78
Jenkins image by Jenkins project is licensed under a Creative Commons Attribution-‐‑‒ShareAlike 3.0 Unported License.
Copyright © NTT Communications Corporation. All rights reserved.
Thanks for your attention
Enterprise Cloud
1st SessionAutomate deployment & benchmark for your OpenStack with Chef, Cobbler and Rally
2nd SessionNFV Service Federation across Heterogeneous Sites
3rd SessionEnhancement on OpenStack Networking for Carrier Cloud Platform
4th SessionOpenStack x Enterprise : Formula for Developing OpenStack-‐‑‒Based Cloud for Enterprise
DevOps
Network
Enterprise