Decreasing false positives in automated testing
Click here to load reader
-
Upload
sauce-labs -
Category
Engineering
-
view
1.777 -
download
2
Transcript of Decreasing false positives in automated testing
Reducing False
Positives
&
Presents
QASource Confidential
QASource & automation
100,000+ automated test cases
100+ Selenium engineers
500+ years collective automation experience
100+ automation frameworks
QASource Confidential
Question #1
How many years of experience do you have with Selenium?
< 2 years 2-4 years 5+ years
What is the size of your automation suite?
< 100
test cases
100-250
test cases
250-500
test cases
500+
test cases
What is the most important reason why you invest in automation?
Save MoneySave Time/
Release Faster
More QA
Coverage
Question #2
Question #3
Attendees poll
QASource Confidential
Anand Ramakrishnan
QA Director,
QASource
• 15+ years of automation experience
• Directing a team of 150 engineers
• M.S. in Computer Applications
• M.S. in Cyber Law & Security
QASource Confidential
Key learnings & takeaways
What are false positives and why do they occur
Common causes and the challenges
How to implement key strategies to reduce them
Case studies and real world examples
FREE step by step guide to creating CSS locators
QASource Confidential
What are false positives?
Tests that are marked as
failed, when in reality they
should have passed or should
functionally work.
QASource Confidential
Why do false positives occur?
Automation
ApproachImplemented
FrameworkWritten Test
QASource Confidential
Relying on UI
Relying on sequence of execution
Manual intervention
Slow performance
Reference data change
Shared environment
Change in UI element properties
Potential causes for false positives
QASource Confidential
Impacts on automation
Ignoring failures
Wasting time investigating
failuresDecrease in productivity
Frustration for engineering team
Risk of missing potential bug
Increasing cost of automation
Uncertainty of application health
QASource Confidential
Attendees poll
What is the rate of false positives
in your automation?
5% or less 5% to 15% 15% to 20% 20% or more
QASource Confidential
Complex workflow application
Rich interactive UI
2500+ automated test cases
Page object pattern approach
Multiple OS, browsers and environments
Case study
QASource Confidential
Frequent UI Changes
Synchronization issues
Memory leak issue
Challenges
Elements identified using
Xpath
Hardcoded waits
No tear down
ROOT CAUSE
CH
ALLEN
GES
QASource Confidential
Keys to reducing false positives
Deploy application on optimal
configurations
QASource Confidential
Keys to reducing false positives
Controlled environment for automation
QASource Confidential
Keys to reducing false positives
Keeping tests short
QASource Confidential
Keys to reducing false positives
Keeping tests independent
TEST SCRIPT 1
TEST SCRIPT 2
TEST SCRIPT 3
TEST SCRIPT 4
Create User
Create Group
Add User to Group
Search User
QASource Confidential
Using right locators for objects identification
Keys to reducing false positives
IDs
Name
CSS locators
QASource Confidential
Bad Practice
Object identification strategy
Good Practice
QASource Confidential
Keys to reducing false positives
Tear-down approach
QASource Confidential
Bad Practice
Tear down approach
QASource Confidential
Good Practice
Tear down approach
QASource Confidential
Keys to reducing false positives
Dynamic object synchronization
QASource Confidential
Bad Practice
Object synchronization
QASource Confidential
Good Practice
Object synchronization
QASource Confidential
Keys to reducing false positives
Re-execution capability
RE-EXECUTION
LOGIC FLAG
TEST SUITE
TEST CASE 1
TEST CASE 2
—————
—————
—————
TEST CASE N
Pass
Failed
RE-EXECUTION OF
FAILED TESTS
Test case
name
Status Re-
executed
TEST CASE 1PASS NO
TEST CASE 2 PASSYES
TEST CASE 3 FAIL YES
_____ _____ _____
TEST CASE N PASS YES
EXECUTION
EXECUTION REPORT
QASource Confidential
Conclusion
Using right locators for objects identification
Dynamic object synchronization
Re-execution capability
Tear-down approach
Keeping tests short Keeping tests independent
Controlled environment for automation
Deploy application on optimal configuration
QASource Confidential
Benefits of eliminating false positives
Certainty of application health
Increase in productivity
Save time not investigating false
positives
Will not miss potential bugs
Decrease cost of automation
QASource Confidential
Get your FREE Step by Step
Guide to Create CSS
Locators
qasource.com/[email protected]
QASource Confidential
Q&A
QASource Confidential