Building Trust and Efficiency in Your Test Suites

www.unicomlearning.com

Next Gen Testing

Summit-201414th Nov. 2014, PUNE

BUILDING TRUST AND EFFICIENCY

IN AUTOMATED TEST SUITES

Krishna Kumar (KK) Sure

Lead Consultant

www.nextgentesting.org

http://www.unicomlearning.com/

http://www.nextgentesting.org/


UNICOM Presents

Next Gen Testing Summit-2014www.nextgentesting.org

First things first…




Why Do We Automate?

UNICOM Presents


Saves Time

and Money

Improves

Accuracy

Increase Test

Coverage

Simulate Huge

Loads Easily

Catches

Problems

Quickly

Enables

Continuous

Delivery




More Around Test Automation

UNICOM Presents


Don’t consider test

automation a

specialization

Test Automation

should be integral part

of your software

development

Software development

team should share

responsibility of

building and

maintaining a robust

suite of tests




UNICOM Presents


Part I:

Trustworthiness




Non-Idempotent Tests/Suites

UNICOM Presents





Issues with Non-Deterministic Tests

UNICOM Presents


Raises False

Alarms

Brings Huge

Maintenance

Costs

Makes It Easy

To Ignore The

Tests

Lowers

Confidence In

Making

Changes

Lowers Trust

In Test suite

Disrupts

Continuous

Delivery




UNICOM Presents


How to build trust back in your

suite




Quick Fix

UNICOM Presents


• Identify non-deterministic tests

• Quarantine them (Comment

them out)




UNICOM Presents


Reasons for non-determinism in tests




Interdependent Tests

UNICOM Presents


sceanrio1 ‘verify consumer can login’

consumer = create_consumer(email, name, password)login_as(consumer)verify_login_successful_for(consumer)

end

sceanrio2 ‘verify consumer creation’

consumer = create_consumer(email, name, password)verify_consumer_count(1)

end




Interdependent Tests…

UNICOM Presents


• One test execution effects the result of another

test

• Sequence of the tests matters

Ways to fix:

• Rebuild your starting state for each test.

• Make sure each test cleans up after itself




Interdependent Tests…

UNICOM Presents


sceanrio1 ‘verify consumer can login’

consumer = create_consumer(email, name, password)login_as(consumer)verify_login_successful_for(consumer)

end

before_scenariodelete_all(consumers)

end

sceanrio2 ‘verify consumer creation’

consumer = create_consumer(email, name, password)verify_consumer_count(1)

end




Asynchrony

UNICOM Presents


sceanrio1 ‘verify consumer can login’consumer = create_consumer(email, name, password)login_as(consumer)verify login_successful_for(consumer)

end

login_successful_for(consumer)sleep 2message_present(‘Welcome consumer’)

end




Asynchrony…

UNICOM Presents


• Slows the tests

• Gives False failures

Ways to fix:

• Poll for a specific change

• Use methods to wait for jQuery property




Asynchrony…

UNICOM Presents


login_successful_for(consumer)wait_for_text_to_be_present(‘Welcome’)message_present(‘Welcome consumer’)

end

start_time=Time.now

wait_for_text_to_be_present(text)while(!text.present)

if (Time.now – Start_time > timeout)exit

endsleep 1

endend




Asynchrony…

UNICOM Presents


wait_for_ajax(text)while(!page.evaluate(‘jQuer.active’).zero?)

if (Time.now – Start_time > timeout)return false

endsleep 1

endend




Dependency on External Systems

UNICOM Presents


Web

Server

App

Q

U

E

U

E

Payment

Gateway

DB




Dependency on External Systems…

UNICOM Presents


• Services can go down without notice

• Services could be slow and hence increase

test execution time

• Test instances of these dependencies might

not be stable

Ways to fix:

• Use test doubles e.g. Stubs, Mocks, Spies




Too Much Automation

UNICOM Presents


• Automating complex test scenarios which don’t

provide enough value

• High chances of failure and hence increases

maintenance costs

Ways to fix:

• Evaluate ROI before automating test scenarios

• Avoid automating scenarios that are complex

but provide less value




Too Much Automation…

UNICOM Presents


Value

Complexity

Quick Wins, High Value, Definitely

Automate

High value, But complex. Priority 2.

High Complexity, low value. Definite No.

Low value, low complexity. Automate,

maybe




Retry Mechanism

UNICOM Presents


• Collect the failures of the test suite execution

• Rerun these failed tests for double checking

First pass:

bundle exec rake test –collect_failures –f failures.txt

Second pass:

bundle exec rake test failures.txt




UNICOM Presents


Part II:

Efficiency




Issues with Inefficient Test Suites

UNICOM Presents


High

Execution

Times

Test

Duplication

Maintenance

Costs




UNICOM Presents


Building efficiency in your test

suite




Useless Tests

UNICOM Presents


• You could accumulate obsolete tests over a

period of time

• Some throw away tests stay for ever

• Multiple tests testing same code paths

Ways to fix:

• Use ATDD, BDD techniques

• Have test refactoring cycles to consolidate and

delete tests

• Integrate code coverage tool




Code Coverage Tools

UNICOM Presents





Code Coverage Tools…

UNICOM Presents





Long Build Times…

UNICOM Presents


• Large number of tests

• Huge build times and hence long feedback

cycles

• Non-deterministic failures




Long Build Times…

UNICOM Presents


Ways to fix:

• Parallelization

• Isolate database

• Isolate external dependencies

• Collect the failures and rerun

• Use headless browsers (phantomjs,

nightmarejs)

• Segregate Sanity and Regression test suites




Testing Everything Everywhere

UNICOM Presents


• Segregated development and test teams

• Testing same code paths multiple times

• Difficulty in maintenance




Testing Everything Everywhere…

UNICOM Presents


Test pyramid

Test refactoring




Key Takeaways

UNICOM Presents


• Test Automation is an integral part of Software

Development

• Share responsibility of test automation as a

team

• It is important to build trust and efficiency in

your test suite to make it effective




Key Takeaways…

UNICOM Presents


• Isolate tests from each other

• Don’t use explicit wait, instead use wait for

conditions

• Reduce dependencies on external systems

• Don’t automate everything. Evaluate ROI

• Put a retry mechanism to get better results




Key Takeaways…

UNICOM Presents


• Consolidate and delete obsolete tests in a

timely manner

• Follow ATDD, BDD approach

• Integrate Code Coverage tool

• Parallelize the test execution to reduce build

time

• Try to stay close to Test pyramid




Thank You

@kksure

[email protected]


Building Trust and Efficiency in Your Test Suites

Technology

Transcript of Building Trust and Efficiency in Your Test Suites