NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland •...
Transcript of NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland •...
![Page 1: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/1.jpg)
Budapest, 26-28 October 2016
NEXT GENERATION TEST AUTOMATIONPresented by Stephan Schulz, CTO of Conformiq
© All rights reserved
![Page 2: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/2.jpg)
About Conformiq
2
• Founded in 1998 in Finland• Software & support services• Conformiq 360° Test Automation• 3rd generation of software• General purpose test design
and automation tools• Worldwide locations• San Jose, CA, USA (HQ)• Helsinki, Finland (R&D)• Bangalore, India (R&D)• Paris, Hyderabad, Toronto (sales & support)
• Partner with SDLC tool providers and testing services companies
• Customers worldwide• Fortune 100 Enterprise & Networking markets• BFSI, Telecom service providers
© All rights reserved
CIO Review20 Most Promising Software Testing Solution Providers 2015
2015TOP20
![Page 3: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/3.jpg)
Food for Thought
• What percentage of tests do you execute today in an automated way? Regression versus progression?
• What is your current requirement coverage?
• How confident are you that all of your regression tests actually need to be executed (at all)?
3
![Page 4: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/4.jpg)
Data from a Previous Poll (Enterprise IT ‐ EU)
• What percentage of tests do you execute today in an automated way?
• What is your current requirement coverage?
4
More than 81% 28%
Between 61% and 80% 35%
Less than 60% 28%
Not sure 10%
More than 80% 10%
Between 41% and 80% 12%
Between 20% and 40% 15%
Less than 20% 56%
We do not automate 7%
![Page 5: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/5.jpg)
Some Related Quotes
• “We have 3 testers per developer and defects still slip through.” Cardinal Health Test Lead
• “I have 30,000 test cases and my testers can not tell me what they cover.” Huawei Test Director
• “I was surprised about the throughput of one of my testers – until I decided to review his test scripts …”Enterprise IT Test Manager
5
![Page 6: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/6.jpg)
In a Perfect World: What would I want?
Increase productivity
Increase and optimize test coverage
Full and complete integration
6
+
100%
![Page 7: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/7.jpg)
It Takes More than Automating Test Execution
7
Define tests
Write scripts
Execute tests
NextIteration
Challenges • Deployment cost• Required skill set• Maintenance• Test quality• Cycle time• Continuous integration
Documentation ScriptsProduct changes
Requirements
![Page 8: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/8.jpg)
Evolution of Software Testing
8
Manual Testing
Test ScriptsCapture/Replay
Test FrameworksKeyword Driven
Test Models
MBT
Impr
oved
Tes
ting
Effi
cien
cy
![Page 9: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/9.jpg)
Combine MBT with Test Execution Automation
9
Manual Testing
Test ScriptsCapture/Replay
Test FrameworksKeyword Driven
Test Models
MBT
Impr
oved
Tes
ting
Effi
cien
cy
Automate test design
Shorten path to test execution automation
Simplify MBT adoption
![Page 10: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/10.jpg)
What is Model Based Testing?
• Capture operation of application in a model instead of writing tests• Generate tests and test data based on your targets• Higher coverage & faster test execution from automatic test design• Scripts, documentation, and coverage automatically in sync
10
Tests
Real Functionality
Automaticgeneration
Targets
Model ofFunctionality
![Page 11: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/11.jpg)
Next Generation Test Automation
11
Track requirement changes automatically
Update documentation automatically
Execute tests automatically
Maintain a model and update everything else
automatically
ALM / Test Management
Test Execution
ALM / Requirement Management
Existing Assets
Create models automatically
MBT
![Page 12: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/12.jpg)
Model Based Testing is Disruptive
Functionality vs Path
Application vs Environment
Coverage vs Number of Tests
Abstraction vs Detail
Automated vs Manual Design
Flexible vs Fixed Test Sets
95% 42
Test 1: …Test 2: …Test 3: …
Test 1: …Test 2: …Test 3: …
![Page 13: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/13.jpg)
Complete Automation of End2End Testing Process
13
ModelingManual Tests
GherkinFlowcharts
BPMApp
UnderTest
Test Execution
Test Management
Requirement Management
VersionControl
Test Design
AutomationAssets
![Page 14: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/14.jpg)
Model Capture from Existing Assetssub-diagram
activity
decision
test data
• Model requirements in workflows to be tested
• Import Gherkin feature files or manual tests from spreadsheets
• Import models from flowchart or other BPMN tools
• Model across all interfaces available for testing
• Import interface information from UI recordings, API specs (WSDL/XSD, WADL/JSON, …)
event
1
![Page 15: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/15.jpg)
Review Automatically Generated Tests
• Validate assumptions with all stakeholders before test execution• Know what you cover … and what you don’t!
15
Generated Tests
Test Steps & Data
Test CoverageTest
Targets
2
![Page 16: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/16.jpg)
Integrate with all [Existing] SDLC Tools
16
Version Control
ALM / Test Management
Automatic Test Execution
ALM / Requirement Management
Existing Tests, Recordings & User Stories
3
Gherkin
MBT
![Page 17: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/17.jpg)
It is not a Dream or Vision ‐ it is Very Real!
• “From 3 Stars Rating to 5 Stars in the App Store” Javier Moscardó ‐ Spotify ‐ UCAAT 2015 (best presentation)
• “Requirements and Challenges with Advanced Test Automation, The Industry Perspective”Sigrid Eldh ‐ Ericsson ‐ UCAAT Keynote 2015
• “The World's Largest MBT Deployment Transforms Testing for Quality at Agile Speed”Clark Cochran ‐ Conformiq ‐ STC 2015
• “Shifting Test Automation to the Left of SDLC”Ish Kumar – Tech Mahindra – Blog 2015
17
![Page 18: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/18.jpg)
What will get us to the “6 star rating”?
• Next step in evolution of automated testing is appearing on the horizon already …• What is likely to fail?• What is failing and why?• Is it a symptom or defect?• And more …
18
![Page 20: NEXT GENERATION TEST AUTOMATION€¦ · About Conformiq 2 • Founded in 1998 in Finland • Software & support services • Conformiq 360° Test Automation • 3rd generation of](https://reader033.fdocuments.us/reader033/viewer/2022043023/5f3f148b4f9bc3476940e842/html5/thumbnails/20.jpg)
BACKUP
© All rights reserved