[QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps
-
Upload
future-processing -
Category
Software
-
view
53 -
download
1
Transcript of [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps
![Page 1: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/1.jpg)
SQA in a TestOps era
![Page 2: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/2.jpg)
SQA in a TestOps era
TomaszJaniszewski
DawidPacia
![Page 3: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/3.jpg)
THE TRADITIONAL MODEL OF
SOFTWARE TESTING
![Page 4: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/4.jpg)
THE TRADITIONAL MODEL OF SOFTWARE TESTING
TESTS
QUALITY ASSESSMENT
RUN TESTS
SYSTEM UNDER TEST
ANALYSISPASS? FAIL?
![Page 5: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/5.jpg)
“To improve is to change; to be perfect is to change often.”
Winston Churchill
![Page 6: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/6.jpg)
➜ Facebook users send on average over 31 million messages and view 2.77 million videos every minute
➜ Twitter users generates 347,222 Tweets each minute – or 21 million Tweets per hour
➜ Google provides search results for over 1 billion search queries each day
➜ Amazon deploys new software to production every 11.6 seconds.
The times has changed...
![Page 7: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/7.jpg)
Let’s start with the firstset of slides
1.TRANSITION
HEADLINE
![Page 8: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/8.jpg)
EXPOSURE CONTROL
TEST CASES AND MONITORS
THE TESTOPS MODEL OF SOFTWARE TESTING
EXPOSED USERS AND SYSTEMS
SYSTEM UNDER TEST
PRODUCTION USERS AND SYSTEMS
ACTIVE MONITORS
ANALYSIS
QUALITY ASSESSMENT
PRODUCTION USAGE
TELEMETRY DATA
KPIs / PATTERNS
![Page 9: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/9.jpg)
CONTINUOUS BUILD
CODE CHANGE / FIX
SCM(Git, Svn, Bazaar ;)
NEW BUILD(Ant, Maven, Gradle, etc.)
CI TOOLS(Jenkins, Bamboo, Travis CI, Ansible,
etc.)
CONTINUOUS VALIDATION
TEST ENVIRONMENTS
BUILD
DEPLOYMENT
TEST RESULTS
CONTINUOUS DELIVERY
PRODUCTIONENVIRONMENTS
CONTINUOUS MONITORING
MONITORING(Kibana, Grafana,
Scalyr, Cacti, Amazon
CloudWatch, etc.)
SYNTHETIC / SEMANTIC
MONITORING
![Page 10: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/10.jpg)
➜ Synthetic monitoring (or semantic monitoring) runs a subset of an application's automated tests against the live production system on a regular basis.
Continuous Monitoring
![Page 11: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/11.jpg)
➜ Integration with DevOps infrastructure
➜ What tests to run and when➜ No need to stand up the whole
system to test
TestOps: What Does This Mean?
![Page 12: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/12.jpg)
WHY SHOULD WE CARE SO
MUCH?
![Page 13: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/13.jpg)
FAKAPY● Zgłaszanie błędu w komentarzy w Jirze - 500 klientom nie działało menu
nawigacyjne po aplikacji● Dobrze zorganizowane glow w Gicie - pull req, praca miedzy zespolami● Szkiletet ● Jak pisac kod (python)● Nazwy obiektow!● Re-uzywalnosc kodu● Korzystanie z paczek● Pop● Pliki readme● Jak unikac zlych praktyk● Konwencja komitowanie (a nie ‘fixed small issue’)● Konsekwencja pracy przez 2-mc na wlasnym branchu● Dlaczego nie tylko produkt powinien byc testowany, ale również same
testy
![Page 14: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/14.jpg)
BUT IN A WORLD OF FALSE...
False Negatives
False Positives
The test is marked as Failed, when the functionality is
working
The test is marked as Passed, when the functionality is not
working.
![Page 15: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/15.jpg)
...YOU CAN ENCOUNTER SOMETHING WORSE
False Negatives
False Positives
Flaky Tests
TESTS FAILING INTERMITTENTLY
![Page 16: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/16.jpg)
RIS
KV
ALU
ECONTINUOUS: VALUE vs RISK
PLAN CODE BUILD TEST RELEASE DEPLOY OPERATE
COLLABORATION
AGILE DEVELOPMENT
CONTINUOUS INTEGRATION
CONTINUOUS DELIVERY
DEVOPS
![Page 17: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/17.jpg)
Dawid Pacia
“More automatically, more continuously,
more risky...”
![Page 18: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/18.jpg)
TESTOPS TOOLBOX AND BESTPRACTICES
![Page 19: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/19.jpg)
GIT PROPER MODEL
19
time / commits
master
development
feature 1
feature 2
release
The Great QA Wall
![Page 20: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/20.jpg)
![Page 21: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/21.jpg)
Pipelinespipeline { agent none stages { stage('error') { steps { parallel( "Wait": { sleep 1 }, "NodeExec": { node(label: 'master') { isUnix() }… } stage('ExternalJob') { steps { build(job: 'TestMulti1', propagate: true) } ...
Exam
ple
![Page 22: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/22.jpg)
But here it comes… Blue Ocean!
![Page 23: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/23.jpg)
➜ You can use compose to bring up any number of instances of the services in your stack, example:
> docker-compose up --scale web=10 database=3
This is great for testing database clusters or load balancers, or resiliency with Pumba
Docker - Scalability Testing
![Page 24: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/24.jpg)
➜ Docker can be useful for testing different system configurations like for example different DB backends:
○ Bring up new stack (with MySQL backend), Seed with data, Run API tests, Tear down stack.
○ Bring up new stack (with Postgres backend), Seed with data, Run API tests, Tear down stack.
○ Report on any differences in test results between MySQL and Postgres backends.
○ Integrates easily with Jenkins !
Docker - Configuration Testing
![Page 25: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/25.jpg)
Selenium GridRIP
![Page 26: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/26.jpg)
Regression Tests
Jenkins CI
Real-time reporting of Selenium tests on Kibana
UITests
AcceptanceTests
Apache Kafka
Elasticsearch Logstash
Jobs
Logs
Logs
Transformed documents
Kibana
Query results
![Page 27: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/27.jpg)
Final Results
![Page 28: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/28.jpg)
➜ Setup overview○ Configure Apache JMeter (through user.properties file) to
Create a CSV File:
/opt/apache/jmeter/apache-jmeter-3.2/bin/jmeter -n -t Basic\ Elasticsearch\ Test.jmx -l ./results.csv
○ Configure and Run Logstash○ Visualize the Results
Real-time reporting of JMeter results on Kibana
![Page 29: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/29.jpg)
Final Results
![Page 30: [QE 2017] Dawid Pacia, Tomasz Janiszewski - SQA w erze TestOps](https://reader036.fdocuments.us/reader036/viewer/2022062523/5a66aeb77f8b9a21208b4659/html5/thumbnails/30.jpg)
➜ TestOps provides a unique aspect for automated testing
➜ It works with the Dev team to set up CI/CD/CM
➜ It creates a cloud or virtual technology to set up a testing environment on demand
➜ It plays a vital role to integrate the automated testing with Dev team for CI
➜ Explore and investigate!
Takeaway