Wojciech Seliga [email protected], T:...
Transcript of Wojciech Seliga [email protected], T:...
Taming automated testsbeast
2
About me
• 29 years of coding
• Agile Practices (inc. TDD) since 2003
• ScrumMaster, Agile Coach, Trainer, Speaker
• 4.5 years with Atlassian (JIRA Development Team Lead)
• Co-founder of Spartez
3
The Story
4
Codebase10 years old
5
Growing from 2 to about 40 engineers
6
Obsessed with Quality
7
Obsessed with Automated Tests
8
... From the Beginning
9
1.5M lines of code
10
1.5M lines of code
10
1.5M lines of codeCheating
10
Mixture of technologies
11
Mixture of technologies
Java
JSP
Velocity
Webwork
OSGi
Spring
Pico
Soy
OfBiz EntityEngine
QuartzOS Workflow
Jersey
LESS
SeraphJDBC
LuceneJackson
Maven2
JQuery
Backbone.js SpringDM
XML
Active Objects
OAuth
OpenSocial
Javamail
Underscore
REST XStream
Guava
11
DependenciesLots of
12
Dependencies554
mvn dependency:list -DincludeScope=compile -o | grep :jar | cut -c 11- | sed s/:provided// | sed s/:compile//| sort -u|wc -l12
65 modules in one IntelliJ project13
65 modules in one IntelliJ project13
65 modules in one IntelliJ project13
14
13000 unit tests
15
Almost 1000 Selenium Tests
16
4000 Functional and Integration Tests
17
18
Atlassian JIRA
18
Our CI environment
19
Test frameworks
• JUnit 3 and 4
• JMock, Easymock, Mockito
• Powermock, Hamcrest
• QUnit, HtmlUnit
• JWebUnit, Selenium, WebDriver
• Custom runners, extensions, matchers
20
Test frameworks
• JUnit 3 and 4
• JMock, Easymock, Mockito
• Powermock, Hamcrest
• QUnit, HtmlUnit
• JWebUnit, Selenium, WebDriver
• Custom runners, extensions, matchers
20
Bamboo Setup
• Dedicated server with 60+ remote agents (including Amazon Elastic)
• Build engineers
• Bamboo devs
21
22
for each main branch22
for each main branch22
23
Run first
23
Run in parallel in batches
Run first
23
There is
MuchMore
24
Type of Tests
• Unit
• Functional
• Selenium / WebDriver
• Integration
• Platform
• Performance
25
Platforms
26
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
26
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
26
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
26
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
26
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
26
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
• Dimension - Browsers: IE 8+, FF, Chrome,
26
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
• Dimension - Browsers: IE 8+, FF, Chrome,
26
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
• Dimension - Browsers: IE 8+, FF, Chrome,
Coming
26
Platforms
• Dimension - DB: MySQL, PostgreSQL, MS SQL, Oracle
• Dimension - OS: Linux, Windows
• Dimension - Java ver.: 1.5, 1.6, 1.7
• Dimension - CPU arch.: 32-bit, 64-bit
• Dimension - Deployment Mode: Standalone, Tomcat, Websphere, Weblogic
• Dimension - Browsers: IE 8+, FF, Chrome,
Run Nightlyor Before Release
Coming
26
Triggering Builds
• On Commit (hooks, polling)
• Dependent Builds
• Nightly Builds
• Manual Builds
27
But...
28
Slow unit test29
Very slow functional tests30
Builds Wait in The Queue31
Builds Often Fail32
Too Often...33
It takes time to fix it...34
Sometimes very long35
You commit at 3 PM
36
You commit at 3 PM
You get “Unit Test Green” email at 4PM
36
You commit at 3 PM
You get “Unit Test Green” email at 4PM
You happily go home
36
You commit at 3 PM
You get “Unit Test Green” email at 4PM
You get flood of “Red Test X” emails at 4 - 9PM
You happily go home
36
You commit at 3 PM
You get “Unit Test Green” email at 4PM
You get flood of “Red Test X” emails at 4 - 9PM
Your colleagues on the other side of the globe
You happily go home
36
You commit at 3 PM
You get “Unit Test Green” email at 4PM
You get flood of “Red Test X” emails at 4 - 9PM
Your colleagues on the other side of the globe
You happily go home
You36
“Slow CI loop and non-deterministic tests are
strong inhibitor of change instead of the catalyst”
by W. Seliga37
“We probably spend more time dealing with the JIRA
test codebase than the production codebase”
38
0
25
50
75
100
Test Coverage
Effort Invested
Striving for Coverage
39
0%
25%
50%
75%
100%
Value
Size of automated tests
Strange? Relationship
40
Test Hell - The Outcomes
• Development slows down
• Devs are afraid of change
• Software difficult to release
• Significant amount of time spent on analysing test failures
• Morale goes down
41
FeedbackSpeed
`Quality
42
Feedback Loop Speed
• Tiniest change triggers test avalanche
• Lack of responsibility syndrome
• Devs do not run tests locally (speed)
• Before you get the results you are at home
43
Quality
• Non-deterministic tests (races, timeouts)
• Catching up with UI changes
• 1 red test hides new failures
• Ignoring always red tests in dangerous ...
44
Broken window theory45
Decisions which do not scale
46
Decisions which do not scale
• All unit tests in one maven module
46
Decisions which do not scale
• All unit tests in one maven module
• All functional tests in one maven module
46
Decisions which do not scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one module
46
Decisions which do not scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one module
• Every commit triggers rebuild and re-test of everything
46
Decisions which do not scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one module
• Every commit triggers rebuild and re-test of everything
• Monolithic test framework / utils
46
Decisions which do not scale
• All unit tests in one maven module
• All functional tests in one maven module
• All Selenium and web-driver tests in one module
• Every commit triggers rebuild and re-test of everything
• Monolithic test framework / utils
• Opaque fixtures
46
Strategies
47
Test Quality
48
Catching up with UI changes
Problem:
49
Catching up with UI changes
Problem:
Solution:
49
Catching up with UI changes
Page Objects Pattern
Problem:
Solution:
49
Page Objects Pattern• Page Objects model UI elements (pages,
components, dialogs, areas) your tests interact with
• Page Objects shield tests from changing internal structure of the page
• Page Objects generally do not make assertions about data. The can assert the state.
• Designed for chaining50
Page Objects Examplepublic class AddUserPage extends AbstractJiraPage{
private static final String URI = "/secure/admin/user/AddUser!default.jspa";
@ElementBy(name = "username") private PageElement username;
@ElementBy(name = "password") private PageElement password;
@ElementBy(name = "confirm") private PageElement passwordConfirmation;
@ElementBy(name = "fullname") private PageElement fullName;
@ElementBy(name = "email") private PageElement email;
@ElementBy(name = "sendemail") private PageElement sendEmail;
@ElementBy(id = "user-create-submit") private PageElement submit;
@ElementBy (id = "user-create-cancel") private PageElement cancelButton;
@Override public String getUrl() { return URI; }
...
@Override public TimedCondition isAt() { return and(username.timed().isPresent(), password.timed().isPresent(), fullName.timed().isPresent()); }
public AddUserPage addUser(final String username, final String password, final String fullName, final String email, final boolean receiveEmail)
{ this.username.type(username); this.password.type(password); this.passwordConfirmation.type(password); this.fullName.type(fullName); this.email.type(email); if(receiveEmail) { this.sendEmail.select(); } return this; }
public ViewUserPage createUser() { return createUser(ViewUserPage.class); }
public <T extends Page> T createUser(Class<T> nextPage, Object...args) { submit.click(); return pageBinder.bind(nextPage, args); }
51
Using Page Objects @Test public void testServerError() { jira.gotoLoginPage().loginAsSysAdmin(AddUserPage.class) .addUser("username", "mypassword", "My Name",
"[email protected]", false) .createUser();
// assertions here }
52
Using Page Objects @Test public void testImportSampleProject() { final PivotalImporterSetupPage setupPage = getSetupPage(); assertEquals("1. Connect", setupPage.getActiveTabText());
final PivotalProjectsMappingsPage projectMappingPage = setupPage.next(); assertEquals("2. Project Mapping", setupPage.getActiveTabText()); assertTrue("Expecting all project to be selected by default", projectMappingPage.areAllProjectsSelected()); projectMappingPage.setImportAllProjects(false); projectMappingPage.setProjectImported(sampleProject, true); projectMappingPage.createProject(sampleProject, sampleProject, "SAMPLE");
final ImporterFinishedPage importerLogsPage = projectMappingPage.beginImport().waitUntilFinished(); assertTrue(importerLogsPage.isSuccess()); assertEquals(0, importerLogsPage.getGlobalErrors().size()); assertEquals("1", importerLogsPage.getProjectsImported()); // more assertions here }
53
More on Page Objects
• Design for reusability
• Design for sharing - libraries of Page Objects
• Good support by WebDriver/Selenium 2
• Atlassian Selenium 2.0
54
Page Objects Caveat
The easier is to write functional tests, the more tests will be written...
The more stuff you will need to maintain,the more time the execution of your tests will take...
W. Seliga, PO. 1-4
55
Opaque Test FixturesProblem:
56
Opaque Test FixturesProblem:
Solution:
56
Opaque Test Fixtures
REST-based Set-up,Backdoors
Problem:
Solution:
56
REST-based Setup
57
REST-based Setup @Before public void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml"); }
57
REST-based Setup @Before public void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml"); }
VS
57
REST-based Setup @Before public void setUpTest() { restore("some-big-xml-file-with-everything-needed-inside.xml"); }
@Before public void setUpTest() { restClient.restoreEmptyInstance(); restClient.createProject(/* project params */); restClient.createUser(/* user params */); restClient.createUser(/* user params */); restClient.createSomethingElse(/* ... */); }
VS
57
Flakey TestsProblem:
58
Flakey TestsProblem:
Solution:
58
Flakey Tests
Quarantine
Problem:
Solution:
58
Flakey Tests
Quarantine
Problem:
Solution:
Fix58
Flakey Tests
Quarantine
Problem:
Solution:
Fix Eradicate58
Quarantine• @Ignore, @Category
• Quarantine on CI server
• Recover or Die - limiting quarantine
59
Quarantine• @Ignore, @Category
• Quarantine on CI server
• Recover or Die - limiting quarantine
59
Quarantine• @Ignore, @Category
• Quarantine on CI server
• Recover or Die - limiting quarantine
59
Heal or Kill
60
Heal or Kill
60
Quickly!
Fixing Flakey TestsProblem:
61
Fixing Flakey TestsProblem:
Solution:
61
Fixing Flakey Tests
Timed Conditions
Problem:
Solution:
61
Fixing Flakey Tests
Timed Conditions
Problem:
Solution:
Test-friendly Markup61
Fixing Flakey Tests
Timed Conditions
Problem:
Solution:
Mock Unreliable DepsTest-friendly Markup
61
Timed Conditions
62
Test-friendly Markup
• Do not save on IDs
• Do not save on CSS classes
• XPath is fragile
• XPath is expensive
• XPath is not readable
• i18N
63
Mock Unreliable Dependencies
64
Speed
65
SpeedAiming at 10 seconds build
65
Splitting CodebaseThe easiest and most effective
improvement
66
Splitting Codebase
• Tests closer to tested code
• Less to test
• Testing less frequently
• Increased team responsibility
• Restructuring CI hierarchy - more complicated picture
67
Less to Test
68
Less to TestCommit
68
Less to TestCommit
68
Less to TestCommit
68
Less to TestCommit
68
Less to Test
68
Less to Test
Most of commits happen here68
Speed vs Control Workspace Dilemma
• Incubation
• Maturity
• Custom workspaces
69
Test Execution Time
70
Execution Time:Test Level
71
Execution Time:Test Level
Unit Tests
71
Execution Time:Test Level
Unit Tests
REST API Tests
71
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
71
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
71
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
71
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
Speed
71
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
Speed
71
Execution Time:Test Level
Unit Tests
REST API Tests
JWebUnit/HTMLUnit Tests
Selenium/WebDriver Tests
Speed Confidence
71
Test Pyramid
Unit Tests
REST / HTML Tests
Selenium
72
Test Pyramid
Unit Tests
REST / HTML Tests
Selenium
72
Execution time - Cont.
• Batching
• Several tests per single set-up (violation of test isolation)
• REST-based setups and assertions
• Remove / merge overlapping tests
73
Execution time - Cont.
• IDs over CSS/JQuery Selectors over XPath
• JUnit tests running in the container
• In-process testing
• In-memory DBs
• Mocking web servers
• Reducing framework initialization time
• Test Optimization (Clover)
74
Waiting time
• More build agents
• Shorter and smaller tests
• No sleep()
• Avoiding long fixture setup (hot container, fragmented setup)
75
Preparation Time
• SCM performance
• Container set-up
• Compilation time (GWT...)
• Maven...
• Artifacts passing
76
So how about the goals?
Is 10 seconds build realistic?
77
Realistic Goals (for us*)
78
Realistic Goals (for us*)
*My current personal dreams78
Realistic Goals (for us*)
Time Type
2 min for unit tests for 95% of the commits
5 min for base smoke functional tests for 95% of the commits
15 min for ALL tests for 95% of the commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams78
Realistic Goals (for us*)
Time Type
2 min for unit tests for 95% of the commits
5 min for base smoke functional tests for 95% of the commits
15 min for ALL tests for 95% of the commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams78
Realistic Goals (for us*)
Time Type
2 min for unit tests for 95% of the commits
5 min for base smoke functional tests for 95% of the commits
15 min for ALL tests for 95% of the commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams78
Realistic Goals (for us*)
Time Type
2 min for unit tests for 95% of the commits
5 min for base smoke functional tests for 95% of the commits
15 min for ALL tests for 95% of the commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams78
Realistic Goals (for us*)
Time Type
2 min for unit tests for 95% of the commits
5 min for base smoke functional tests for 95% of the commits
15 min for ALL tests for 95% of the commits on selected platform
30 min for ALL tests for ALL commits
*My current personal dreams78
Realistic Goals (for us*) p.2
79
Realistic Goals (for us*) p.2
*My current personal dreams79
Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams79
Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams79
Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams79
Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams79
Realistic Goals (for us*) p.2
Metric Type
>98% green unit tests
>95% green functional tests
<20min average time to fix unit test
<2h average time to fix functional test
*My current personal dreams79
Our Possible Future
• Further splitting the code-base, incubation and maturity
• Finer-grained team responsibilities
• Merciless quarantine and purging of flakey tests
• More page objects, less old-school Selenium
• Refactoring/removal of slow tests
• More REST-driven test fixtures and assertions
• More developer's awareness, training, coaching
80
Take-aways
81
Automated testing has cumulative benefits
82
Automated testing has cumulative benefits
...and cumulative cost
82
Splitting codebase and favouring unit tests
are key aspects of short test feedback loop
83
Test Code is Not Trash
84
Test Code is Not Trash
Respect
84
Test Code is Not Trash
DesignRespect
84
Test Code is Not Trash
Design
Maintain
Respect
84
Test Code is Not Trash
Design
MaintainReview
Respect
84
Test Code is Not Trash
Design
MaintainRefactor
Review
Respect
84
Test Code is Not Trash
Design
MaintainRefactor
Review
Respect Restructure
84
Test Code is Not Trash
Design
MaintainRefactor
Share
Review
Respect Restructure
84
Test Code is Not Trash
Design
MaintainRefactor
Share
Review
Respect
Discuss
Restructure
84
Test Code is Not Trash
Design
MaintainRefactor
Share
Review
Prune
Respect
Discuss
Restructure
84
Optimum Balance
85
Optimum Balance
Isolation85
Optimum Balance
Isolation Speed85
Optimum Balance
Isolation Speed Coverage85
Optimum Balance
Isolation Speed Coverage Level85
Optimum Balance
Isolation Speed Coverage Level Structure85
Optimum Balance
Isolation Speed Coverage Level Structure Effort85
Dangerous to temper with
86
Dangerous to temper with
Quality / Determinism86
Dangerous to temper with
MaintainabilityQuality / Determinism86
87
There are no universal rules - silver bullets
87
There are no universal rules - silver bullets
We are expected to find optimum balance for our
specific case
87
There are no universal rules - silver bullets
We are expected to find optimum balance for our
specific case
Definition of “optimum” constantly changes
87
Otherwise
88
Otherwise
89
Did I mention that Page Objects pattern?
90
We are hiring in Gdańsk!
Java devs Javascript devs UI/UX designers
Team leader
Do you want to help us?
91
Credits
• Photos:• http://www.flickr.com/photos/toofarnorth/ - Dragon
• http://www.flickr.com/photos/striatic - Frustration
• http://www.flickr.com/photos/leeadlaf/ - Broken window
• http://www.flickr.com/photos/johnloo/ - Lightbulb
• Dariusz Kordoński - for Test Improvements Leadership in JIRA
92
Thank You
93