for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This...
Transcript of for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This...
![Page 1: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/1.jpg)
Upstream CI/CD toolsfor your production
clouds
That can operate 1,000+ test cycles in a day
Slide URL: https://goo.gl/aZ3Tq3
![Page 2: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/2.jpg)
Masayuki Igawa
Core in Tempest, openstack-health, subunit2sql
2
![Page 3: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/3.jpg)
Ghanshyam Mann
Core in Tempest
3
![Page 4: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/4.jpg)
Ken’ichi Ohmichi
QA PTL, Core in Nova
4
![Page 5: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/5.jpg)
Agenda
1. Introduction2. Overview of QA tools3. How to use QA tools for productions4. Future works5. Demo
5
![Page 6: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/6.jpg)
Introduction
Why we are here
● We are working for OpenStack quality as upstream QA members● There are many tools for testing/debugging for the quality
What will you learn
● How to keep/improve the quality at upstream development● How to use these tools for production clouds
6
![Page 7: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/7.jpg)
What’s the QA in OpenStack?
7
![Page 8: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/8.jpg)
Mission Statement of QA project
Develop, maintain, and initiate tools and plans to ensure the upstream stability and quality of OpenStack, and its release readiness at any point during the release cycle.
8
![Page 9: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/9.jpg)
Overview of gate testsEvery proposed code needs to be passed via multiple gate tests
● pep8 (coding style)● python 27/34 (unit test)● gate-tempest-dsvm (integration test)
1,000+ gate test cycles run in most working days
● Each gate test cycle consists of 1,000+ tests● That means 1,000,000+ tests run in a day
9
![Page 10: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/10.jpg)
Overview of upstream CI/CDTesting the proposed code on temp env with the CI/CD
devstack-gate Virtual machine
Node pool
devstack Tempest
logs
gerritdeveloper
1. Propose code
2. Kick4. Kick
3. Allocate from pool5. Configure
6. Deploy with proposed code
7. Test
8. Store result and logs9. Developer checks result on gerrit
QA projects 10
![Page 11: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/11.jpg)
Overview of QA tools
11
![Page 12: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/12.jpg)
QA tools
12
Testing: Tempest, os-testr, Grenade, Bashate, Hacking
OpenStack specific: OpenStack-health, subunit2sql, elastic-recheck, DevStack, stackviz
Generic: ElasticSearch/Logstash/Kibana, Gerrit, Jenkins, Graphite
https://wiki.openstack.org/wiki/QA
![Page 13: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/13.jpg)
QA tools
13
Run tests: TempestChecks Results: OpenStack Health Dashboard
Debug Failure: ElasticSearch/Logstash/Kibana
![Page 14: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/14.jpg)
Tempest - OpenStack Integration Test Suite
● This is a set of integration tests to be run against a live OpenStack cluster.● Tempest is designed to be useful for a large number of
different environments. This includes being useful for gating commits to OpenStack core projects and used for testing the production Cloud.
14
![Page 15: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/15.jpg)
Tempest - Test types● API tests
○ API tests are validation tests for the OpenStack APIs.○ Make sure stability and backward compatibility of APIs.
● Scenario tests○ Scenario tests are complex “through path” tests for OpenStack
functionality.○ Scenario tests interact with many services and tests cross projects
features.● Stress tests - Deprecated, will be removed in Newton
○ Stress tests are designed to stress an OpenStack environment by running a high workload against it and seeing what breaks.
15
![Page 16: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/16.jpg)
Tempest - External Plugin
● Tempest provides an external test
plugin interface to enables projects
to implement tempest like tests and
run those as part of tempest run.
~30 plugins
16
![Page 17: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/17.jpg)
OpenStack-Health is a dashboard for visualizing test results of OpenStack CI jobs.
17http://status.openstack.org/openstack-health/
![Page 18: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/18.jpg)
What is the OpenStack-Health?
18
Why do we need this?
BigTents, numerous projects and test jobs, notify to ML doesn’t work
Features
Provide a dashboard :Graphs, Colorful table for analyzing our “BIG-DATA”
Technically
OpenStack-HealthAngularJS 1(FrontEnd) + Flask(API server) subunit2sql API ++ Database
![Page 19: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/19.jpg)
● Context color table: easy to understand the status● RSS feed: Alternative notification● Graph: “Gauge -> Bar” : space effective
OpenStack-Health: Recent Topics
19
![Page 20: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/20.jpg)
OpenStack-Health: Recent Topics (cont.)
Recent Topics
● Regex Search filter: specify multiple projects (e.g. nova|glance|cinder…)● Pagination
20
![Page 21: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/21.jpg)
OpenStack-Health - Data flow
21
DataBasesubunit2sql
Subunit stream(Tempest, unit tests, etc.)
Gearman worker
O-H API
server
subunit2sql
![Page 22: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/22.jpg)
Kibana / Elasticsearch / LogstashLog management system
We are using them for searching similar cases to debug gate problems
debugger
Kibana (logstash.openstack.org)
a) Search cases
Elasticsearch
Logstash
Gearmanworker
1. Fetch logs of gate test and send them to Logstash
2. Filter unnecessary logs out and convert into useful format
3. Store logs with indexing
b) Search logs
22
![Page 23: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/23.jpg)
How to use QA tools for productions
23
![Page 24: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/24.jpg)
How to use Tempest
2 Ways to Use Tempest
24
Tempest as a System Installed
Program
Tempest Source Code Repo
![Page 25: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/25.jpg)
How to use Tempest(cont.)
Install Tempest as system program:
25
$ git clone http://git.openstack.org/openstack/tempest$ pip install tempest/
/etc/tempest<prefix>/etc/tempest
![Page 26: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/26.jpg)
How to use Tempest(cont.)
Set up separate working directory
26
$ tempest init Cloud_01_Test $ tempest init Cloud_02_Test
![Page 27: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/27.jpg)
How to use Tempest(cont.)
Tempest Source Code Repo:
27
$ git clone http://git.openstack.org/openstack/tempest$ cd tempest
![Page 28: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/28.jpg)
How to use Tempest: Set up Config File
Tempest Configurations:
28
~80 config options (Tempest & 6 Core Projects.)
+Additional for other projects tests (Tempest tree or in plugin).
![Page 29: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/29.jpg)
How to use Tempest: Set up Config File
Tempest Configuration Sample File.
$ tox –e genconfig
OR $ oslo-config-generator --config-file etc/config-generator.tempest.conf --output-file etc/tempest.conf
Best Reference for tempest.conf: Devstack generated one.
29
![Page 30: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/30.jpg)
How to use Tempest: Set up Config File(cont.)
30
Auth Configuration:● test_accounts_file● Use_dynamic_credentials
○ Admin_username○ Admin_project_name○ Admin_password○ Admin_domain_name○ Create_isolated_networks○ default_credentials_domain_name
● ...
![Page 31: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/31.jpg)
How to use Tempest: Set up Config File(cont.)
31
Identity Configuration:● uri● uri_v3● auth_version● catalog_type● region● v2_admin_endpoint_type● v2_public_endpoint_type● v3_endpoint_type● ...
![Page 32: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/32.jpg)
How to use Tempest: Set up Config File
Example: Compute Configuration● catalog_type● region● endpoint_type
● api_extensions● image_ref● flavor_ref● resize● …..
32
![Page 33: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/33.jpg)
How to use Tempest:
All Set to Hit the Cloud
33
![Page 34: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/34.jpg)
How to use Tempest: Run Tests
$ testr run <test path>tempest.api.compute.servers. test_servers.serversTestJSON. test_create_server_with_admin_password
$ testr run <test idempotent_id> b92d5ec7-b1dd-44a2-87e4-45e888c46ef0
$ testr run <test directory> tempest.api.compute$ testr run --parallel
34
![Page 35: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/35.jpg)
OpenStack-Health - Basic installation
● Setup subunit2sql & Store your test data to your databasePre condition: Setup your Database(e.g. dbname:subunit, username:subunit, pass:pass)
$ git clone git://git.openstack.org/openstack-infra/subunit2sql $ subunit2sql-db-manage --database-connection mysql://subunit:[email protected]/subunit upgrade head $ subunit2sql --database-connection mysql://subunit:[email protected]/subunit SUBUNIT-V2-FILE
35Usage: https://github.com/openstack-infra/subunit2sql#usage
NOTE:* subunit2sql only takes a subunit v2 stream/file. So if you have only v1, you can convert it with using the subunit-1to2.e.g. cat v1-file | subunit-1to2 | subunit2sql --database-connection mysql://subunit:[email protected]/subunit* You may need to integrate subunit2sql with CI tool like Jenkins.
Must ReadREADME.rst
![Page 36: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/36.jpg)
OpenStack-Health - Basic installation● Setup openstack-health $ git clone git://git.openstack.org/openstack/openstack-health API Server:
$ pip install -r requirements.txt (in virtualenv) Frontend:
$ sudo apt-get install nodejs npm nodejs-legacy$ sudo npm -g install npm@2$ sudo npm -g config set prefix /usr/local$ sudo npm -g install npm$ sudo npm -g install gulp$ npm install
● Run (in production) API Server:
$ sudo etc/openstack-health-api.conf /etc/openstack-health.conf$ uwsgi -s /tmp/uwsgi.sock --module openstack_health.api --callable app --http :5000
Frontend:$ gulp prod; (DEPLOY “build/*” to your env such as Apache)
36
Must ReadREADME.rst
[default]db_uri = mysql+pymysql://query:[email protected]/subunit2sqlignored_run_metadata_keys = build_change build_master build_node :
![Page 37: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/37.jpg)
Future Works
37
![Page 38: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/38.jpg)
Future work: Tempest
38
Resource config
Move Common Interfaces to tempest/lib as stable interface
Document for tempest plugin interface
Extend plugin interface
Cover API Microversion tests
...
![Page 39: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/39.jpg)
Future work: OpenStack-Health
39
Improve Data limitation:
Detect/Store infra failure, check queue (not only gate/periodic)
UI improvement:
Improve messy/confusing UI
“I accessed the o-h site, but didn’t understand it. So I don’t use it anymore” :(
Framework update (AngularJS 1 -> 2)
![Page 40: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/40.jpg)
Demo
40
![Page 41: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/41.jpg)
Demo
41
● Tempest● OpenStack-Health
![Page 42: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/42.jpg)
Demo: How to find necessary patches
42
1. Run Tempest2. If failure, find Tempest change log which has added failed test3. Find “Related-Bug” or something on the commit message4. Find the bug report on OpenStack LP5. Find the patch of the component patch which closed the bug
E.g: test_create_servers_on_different_hosts_with_list_of_servers
The bug report: https://bugs.launchpad.net/nova/+bug/1521928
![Page 43: for your production Upstream CI/CD tools clouds · Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest](https://reader035.fdocuments.us/reader035/viewer/2022081409/60868c0b866c4367f4724749/html5/thumbnails/43.jpg)
Useful Links
● #openstack-qa on Freenode● [email protected]
Tempest● https://git.openstack.org/openstack/tempest
OpenStack-Health● http://status.openstack.org/openstack-health/● https://git.openstack.org/openstack-infra/subunit2sql● https://git.openstack.org/openstack/openstack-health● https://bugs.launchpad.net/openstack-health● OpenStack-Health Dashboard and Dealing with Data from the Gate: https://youtu.be/0__tD88oAlo
43