Continuous Performance - Load testing for developers with gatling @ iSense 2016
Continuous Testing 2016
-
Upload
karim-fanadka -
Category
Software
-
view
53 -
download
0
Transcript of Continuous Testing 2016
![Page 1: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/1.jpg)
Continuous Testing2016Karim Fanadka, DevTest Manager
![Page 2: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/2.jpg)
About me: Karim Fanadka
Job: DevTest Manager at HPE Software, 7+
Product: StormRunner Load
Favorite activates: Traveling, Ski, Blogging
Ski – France 2013
Road Trip – Iceland 2015
![Page 3: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/3.jpg)
We are going to talk about:
AgileModern QA
SaaS
Automation
Environments
Hotfixes Continuous Deployment
![Page 4: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/4.jpg)
SaaS – Software as a Service
![Page 5: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/5.jpg)
Before SaaS – On Premise
Software
Users
![Page 6: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/6.jpg)
SaaS – Software as a Service
Data Base
Servers
File System
Firewall
Users SaaS
![Page 7: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/7.jpg)
Which product my team is testing :
StormRunner Load - SaaS product
Performance testing solution that makes it easy to plan, run, and scale web and mobile testing over the cloud.
![Page 8: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/8.jpg)
![Page 9: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/9.jpg)
9
1 Create and run a load test using your favorite scripting tool
2 Load Generators spin up in AWS EC2 or MS Azure
3 Results and app insights are available in real time
The product my team is testing: StormRunner Load
New
Soon
Performance testing solution that makes it easy to plan, run, and scale web and mobile testing over the cloud.
![Page 10: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/10.jpg)
![Page 11: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/11.jpg)
StormRunner Load as a Service
Data Base
Servers
File System
Firewall
Users SaaS Cloud
![Page 12: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/12.jpg)
Agile – Do It Fast
![Page 13: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/13.jpg)
Before Agile - Delivery Timeline
1 or 2 Major releases to production in 1 year
![Page 14: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/14.jpg)
Agile StormRunner Load Delivery Timeline
March April AugustJune Nov Dec
Whats the CodeName of the next release?If you know the answer, I will invite you
to a AlleyCat BubbleBobble Elevator Action Frogger Golden Axe Hang-On
FebruaryDec
CommanderKeen Digger
6 Major releases to production in 1 yearYear: 2015
![Page 15: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/15.jpg)
Agile – The Challenges• Test new features
• Test Regression
• Automation
• Environments
• Continues Integration / Continues Delivery
• Hotfixes
Do it in 1.5 months
• Data Migration (When update SaaS with the newer version)
• Performance Testing
• Security Testing
• Globalization
• Documentation
• And more …
![Page 16: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/16.jpg)
Modern QA Team (DevTest)
![Page 17: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/17.jpg)
What is DevTest? Using top end QA/DEV practices combined with DevOps techniques
QA & DEV Working very close, on the same IDE, code base, and in united process
Automation is a MUST
Manual is for new features / exception flows that are difficult to automate
NO MORE “Deliveries for QA” YES for “Push to Production”
![Page 18: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/18.jpg)
Training“we are testing, and breaking code constantly but we help rebuild it better…….”
![Page 19: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/19.jpg)
Continues Testing
Push new code
Master
Release branch (1.5 months)
Test new codeShort Regression
Push to production
Automation
X Many times a day
Developer Engineer
Test Engineer DevTest Team
Builds in 1 Release (1.5 month): 1,400• Full: 236• Commit: 1,164
![Page 20: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/20.jpg)
Modern QA
DevTest
DevOpsSaaS
Dev
Product
Customer
![Page 21: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/21.jpg)
Automation
![Page 22: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/22.jpg)
Automation
AlleyC
at
Bubble
Comman
der...
Digger
Elevato
r
Frogge
r
Golden
Axe
Hang-O
n
IceClim
ber
JumpM
an0
100
200
300
400
500
600
700
030
80 100 120150 170
200250
300
0 0
75120
147
220
390
450
600650
0 0 0 0
100
200
270 270 270 270
Automation Active Tests - Trend
Client Backend Cloud
85% Coverage
100% Coverage
65% Coverage X 2 Browsers
Builds Number (JumpMan 1.95): 1,617• Number of Full : 427• Number of Commits: 1,190
![Page 23: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/23.jpg)
23
Labels automation coverage SPEC – 66 pages
![Page 24: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/24.jpg)
24
Product Tree – Regression plan
![Page 25: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/25.jpg)
![Page 26: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/26.jpg)
![Page 27: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/27.jpg)
Frameworks:
Frameworks Tests
Client Testing Framework
API Testing FrameworkCloud Framework
Results
![Page 28: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/28.jpg)
28
Bot
![Page 29: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/29.jpg)
29
![Page 30: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/30.jpg)
Testing based analytics from production
![Page 31: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/31.jpg)
API discovery tool
![Page 32: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/32.jpg)
Cost cloud tool
![Page 33: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/33.jpg)
Cloud Testing
![Page 34: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/34.jpg)
Example for Automation challenge17 Regions in the Cloud 15 different metrics
255 Tests
![Page 35: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/35.jpg)
Automation – Cloud Framework - Report
![Page 36: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/36.jpg)
Test Environments
![Page 37: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/37.jpg)
Test Environments Farm
QA Security Performance IntegrationLocal (Linux)
Staging
![Page 38: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/38.jpg)
Hotfixes in 1 day
![Page 39: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/39.jpg)
Hotfixes Lifecycle
07:00Production bug
08:00Research by QA and Dev
10:00Fixing bug
16:00Testing (Staging)
14:00Testing (Local)
18:00 Push to production
![Page 40: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/40.jpg)
Articles • 10 best practices for QA teams to deliver quality software, fast
• Blurring the lines: How the development tester role changes on agile teams
• 3 steps to building test databases for the real world
• Plan your tests and release criteria the right way
![Page 41: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/41.jpg)
Tip
“Changing our mindset takes precedence over technology. When we work together, we will succeed.”
Thank you
![Page 42: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/42.jpg)
@kim0z85
Question?
https://www.facebook.com/karim.f.as
https://il.linkedin.com/in/kareemfa [email protected]@HPE.com
Karim FanadkaDevTest ManagerHPE Software
![Page 43: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/43.jpg)
46
![Page 44: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/44.jpg)
47
12/27/2015 3/6/2016
3/6/2016Go liv e
3/2/2016Staging
2/28/20 16CF
2/14/20 16FF
1/3/2016DevTest Start
Ice Climber V1.9 (Demo T imeline)
• Production
• Staging
• CF
• FF
• Dev
![Page 45: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/45.jpg)
48
CIProductionFFCF StagingSecurity + SAASFeature discovery Defects -> ER’s
![Page 46: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/46.jpg)
In Progress
49
12/27/2015 3/6/2016
3/6/20 16Go liv e
3/2/2016Staging
2/28/20 16CF
2/14/2016FF
1/3/2016DevTest Start
Ice Climber V1.9 (Demo T imeline)
Preparations: Feature discovery (meetings) Defects review (re prioritize, clean) Writing automation spec - example Automation design for exist and new frameworks
Testing: Test new features (Top priority, to make sure no delay in the timeline) Write automation tests to increase coverage (in the first 2 weeks, content is very low)
![Page 47: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/47.jpg)
Feature Freeze
50
12/27/2015 3/6/2016
3/6/20 16Go liv e
3/2/2016Staging
2/28/20 16CF
2/14/2016FF
1/3/2016DevTest Start
Ice Climber V1.9 (Demo T imeline)
10 days for stabilization : Test un finished features (2 days in the FF) Prepare [Cloud deployment, move to Release branch, prepare regression plan, update AGM] Regression ~7 days – Product matrix Migration – Copy DB from production + File system
In addition: Prepare Security Env Demo all the features to Security / SaaS Discover next release features
![Page 48: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/48.jpg)
Code Freeze
51
12/27/2015 3/6/2016
3/6/20 16Go liv e
3/2/2016Staging
2/28/20 16CF
2/14/2016FF
1/3/2016DevTest Start
Ice Climber V1.9 (Demo T imeline)
2 days for final validations : Verify defects that was fixed in the end of FF Validate Security + Performance + Globalization have no holders
Staging ~1 day sanity
Production ~1 day sanity
![Page 49: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/49.jpg)
CI/CD
52
Follow all Jenkins failures Test issue: Fix test Code issue: Contact Dev and fix directly / Open defect CI/CD Env issue: Contact DevOps
Deployment SaaS QA1 / SaaS Q2 / Migration / Security Cloud images
On Premise
Builds Number (JumpMan 1.95): 1,617• Number of Full : 427• Number of Commits: 1,190
![Page 50: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/50.jpg)
Defects
![Page 51: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/51.jpg)
54
Coverage : 65% - example Tests: 700 active tests UI product tree covergae
AutomationFrameworks
Client Backend Coverage: 85% - example API discover
Cloud Coverage: All regions – 20 Regions
Protocols –
Metrics
NV
Save months of WD Cover new LG’s very fast Duration: 1.5 hour
~13 min (all tests)
Commit
Full
~2.5 min (main flows)
![Page 52: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/52.jpg)
Automation
AlleyC
at
Bubble
Comman
der...
Digger
Elevato
r
Frogge
r
Golden
Axe
Hang-O
n
IceClim
ber
JumpM
an0
100
200
300
400
500
600
700
030
80 100 120150 170
200250
300
0 0
75120
147
220
390
450
600650
0 0 0 0
100
200
270 270 270 270
Automation Active Tests - Trend
Client Backend Cloud
85% Coverage
100% Coverage
65% Coverage X 2 Browsers
Builds Number (JumpMan 1.95): 1,617• Number of Full : 427• Number of Commits: 1,190
![Page 53: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/53.jpg)
56
Next slides are reference – DON’T PRESENT
![Page 54: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/54.jpg)
57
Labels automation coverage SPEC – 66 pages
![Page 55: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/55.jpg)
58
Product Tree – Regression plan
![Page 56: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/56.jpg)
59
API Coverage
![Page 57: Continuous Testing 2016](https://reader035.fdocuments.us/reader035/viewer/2022062522/5880db221a28ab9c3a8b7039/html5/thumbnails/57.jpg)
60
API Coverage