Post on 07-Aug-2015
Automated Testing 101
By Tabby Chapman
@tabbymarie
http://www.tabbychapman.com
1
About Tabby Chapman@tabbymarie
https://github.com/tabbymarie
2
About Me
• Work for the University of California, Irvine
• Computer Scientist
• Developer Cheerleader
• Education Junkie
3
About Me
• WordPress Skillz– Reaching my 10th anniversary developing
with WordPress – Started with 2.0.5, building a custom theme.– Ran my own company for four years– Joined Zeek for almost two years– Currently build highly specialized custom
themes and plugins for UCI.
4
About This Presentation
• Geared towards intermediate QA testers and Developers and also for stakeholders/clients/customers
• Assumptions:– You know what WordPress is.– You know what PHP is.– You have written PHP code several times.
Successfully. Or.– You test all the things.
5
About This Presentation
• We’ll go over– The Developer Workflow(s)– Different Types of Tests– Automated Testing– Demo– Last Words
6
How it Currently Works
7
• Generic Software Development Lifecycle– In the dinosaur days of computers, the
majority of its users were very technical people.
– As computers became more accessible (and smaller), more and more users began to use it
– Developers needed to create a process to think outside of their own processes to accommodate non-technical users
How it Currently Works
8
• Generic Software Development Lifecycle (cont)– Many various methods were created to
attempt to meet this need– One of these is the Software Development
Lifecycle (which was a similar process used in other disciplines)
– It is a top down approach: generic to specific– Specification, Design, Coding
How it Currently Works
9
• Generic Software Development Lifecycle (cont)– Waterfall method is created in the ‘70s. – Structured analysis, Structured design,
Structured programming (sounds a lot like specification, design, and coding)
– Most developers tend to use Define, Design, Develop, Deploy, Maintain.
Generic Modern Development
10
Image credit: https://www.owasp.org/images/8/84/SDLC.jpg
Freelance Development Cycle
11
Image credit: http://mohamadsubhan.files.wordpress.com/2009/06/linear.jpg
How it Currently Works
12
• Full Software Development Lifecycle– Many developers on teams tend to use some
form of the full development lifecycle, as there tends to need to be some form of accountability for teams.
– This includes: feasibility study, requirements definition, software design, implementation and unit testing, integration and system testing, operation and maintenance.
Team Development Cycle
13
Image credit: http://www.softwaretestinghelp.com/unit-testing/
Types of Tests
• Black box• White box• Unit • Integration• Functional • System• End-to-end
• Sanity• Regression• Acceptance• Load• Stress• Performance• Usability
14
Source: http://www.softwaretestinghelp.com/types-of-software-testing/
Testing Players
15
• Quality Assurance (QA) Engineers
• Developers
• Product Owners
• Clients
• End-Users
• Other Stakeholders
Black Box Testing?
16
Black Box Testing
17
• Tests against requirements and functionality. Is not concerned with the internal workings.
• “If I put in 2+4, I expect 6. Does that work?”
White Box Testing?
18
White Box
19
• Tests against requirements and functionality with knowledge of the inner workings.
• “I know that I this system has these conditionals, so I will test all possible outcomes.”
Unit Testing?
20
Unit Testing
21
• Tests specific methods or functions, created by a programmer
• “Does function addTwoIntegers($num1,$num2) work as expected?”
Integration Testing?
22
Integration Testing
23
• Testing of whole application after new work has been integrated as they interact with the client/server
• “Does this site’s individual modules still work as expected when we put it all together?”
Functional Testing?
24
Functional Testing
25
• Black Box testing for specific functional requirements.
• “If I tell it to add five text inputs and send an email, does it add five inputs and send an email as expected?”
Systems Testing
26
• Black Box Testing based on overall requirements.
• “Testing All the Things”. “Does this system meet all of the requirements as expected?”
End-to-end Testing
27
• Similar to Systems, but also tests interaction between various elements, protocols, layers and servers.
• “Testing All the Things and Moar”. “Does this system meet all of the requirements as expected and communicate as expected to its various parts?”
Sanity Testing
28
• A pre-test to make sure the new development is done enough to fully test.
• “Does it at least work a little bit?”
Regression Testing
29
• Testing the whole application after something has been modified to see if something was broken as a result of the modification.
• “Does it STILL work as expected?”
Acceptance Testing
30
• User or Customer does a round of manual tests to make sure that it meets their requirements/needs
• “Does the client think it works as expected?”
Stress Testing
31
• Testing Performance Requirements
• “Can more than one interaction happen at the same time? How many more than one?”
Usability Testing
32
• Testing for user-friendliness. This can be done BEFORE and AFTER development phase.
• “Yeah, it works, but is it easy to use?”
Which testing is important for developers?
33
Which testing is important for developers?
• Black box• White box• Unit • Integration• Functional • System• End-to-end
• Sanity• Regression• Acceptance• Load• Stress• Performance• Usability
34
Source: http://www.softwaretestinghelp.com/types-of-software-testing/
For the Developers
35
• Which testing is important for developers?– All of them (know thy tester and thy tester’s
methods)– But especially
• Unit• Functional• Systems• Integration
Which testing is important for testers?
36
Which testing is important for testers?
• Black box• White box• Unit • Integration• Functional • System• End-to-end
• Sanity• Regression• Acceptance• Load• Stress• Performance• Usability
37
Source: http://www.softwaretestinghelp.com/types-of-software-testing/
For the Testers
38
• Which testing is important for testers?– All of them (uphold they developer to unit
tests)– But especially
• Black Box• White Box• Integration• Systems• Sanity
Which testing is important for Clients/Stakeholders?
39
Which testing is important for Clients/Stakeholders?
• Black box• White box• Unit • Integration• Functional • System• End-to-end
• Sanity• Regression• Acceptance• Load• Stress• Performance• Usability
40
Source: http://www.softwaretestinghelp.com/types-of-software-testing/
For the Clients/Stakeholders
41
• Which testing is important for clients/stakeholders?
• Black Box• Usability• Regression• Acceptance
For the Clients/Stakeholders
42
• Which testing is important for clients/stakeholders?
• Black Box• Usability• Regression• Acceptance
What does Automation of Testing mean?
43
What does Automation of Testing mean?
44
• The traditional/easiest way to test what you develop is to make a change, and then load the app to see if that change was effective. Very Manual
• Automated Testing means to write code to test code and use testing software to test the front-end. Don’t rely on the old manual check.
How does this apply to WordPress?
45
• WordPress utilizes a ‘test suite’ “which have multiple test cases and a library and command line tool that runs the test suite or suites” according to WordPress’ automated testing core handbook page
• For this, they use phpUnit. https://phpunit.de
How does this apply to WordPress?
46
• PHPUnit has a series of commands that you can use to run all tests, specific tests, or specific types of tests.
Developer Workflow
47
• When you are ready to write your plugin:– Think of the outcomes you want for your
plugin– Think of the individual methods you want to
write. Should have one output.– Write a test that checks that the output is what
you expect.– Write your method.– Lather. Rinse. Repeat.
Developer Workflow
48
• Testing Tools
• phpUnit (official wordpress test suite)
• WordPress Developer Branch– https://make.wordpress.org/core/handbook/au
tomated-testing/
• Terminal or Editor with Testing Integrated
Developer Workflow
49
Editors (not limited to…)– TextMate - https://macromates.com/download
• (https://github.com/meritt/phpunit-tmbundle)
– Sublime – http://www.sublimetext.com/• https://packagecontrol.io/packages/PHPUnit
– phpStorm - https://www.jetbrains.com/phpstorm/
• Supports all OS’• Phpunit built in
Developer Workflow
50
• Put your tests IN your plugin
• /plugins/[plugindir]/tests/test-[pluginslug].php
• Add a phpunit.xml file:
• /plugins/[plugindir]/tests/phpunit.xml
Developer Workflowphpunit.xml
51
<?xml version=“1.0” encoding=“UTF-8” ?>
<phpunit bootstrap=“vendor/autoload.php” >
<testsuites>
<testsuite>
<directory prefix="test-” suffix=".php">
./tests/
</directory>
</testsuite>
</testsuites>
</phpunit>
Developer Workflowphpunit.xml
52
<?xml version=“1.0” encoding=“UTF-8” ?>
<phpunit bootstrap=“vendor/autoload.php” >
<testsuites>
<testsuite>
<directory prefix="test-” suffix=".php">
./tests/
</directory>
</testsuite>
</testsuites>
</phpunit>
Developer Workflowphpunit.xml
53
<?xml version=“1.0” encoding=“UTF-8” ?>
<phpunit bootstrap=“vendor/autoload.php” >
<testsuites>
<testsuite>
<directory prefix="test-” suffix=".php">
./tests/
</directory>
</testsuite>
</testsuites>
</phpunit>
Developer Workflowphpunit.xml
54
<?xml version=“1.0” encoding=“UTF-8” ?>
<phpunit bootstrap=“vendor/autoload.php” >
<testsuites>
<testsuite>
<directory prefix="test-” suffix=".php">
./tests/
</directory>
</testsuite>
</testsuites>
</phpunit>
Tester Workflow
55
• When you are ready to test – Be familiar with the general requirements for
the site as a whole– Read the requirements of the change request– Focus on specifically what is supposed to
have changed.– Write a test while waiting for the change.– Run the test once you get the change from
the developer
Tester Workflow
56
• Selenium IDE, Firefox Plugin– http://www.seleniumhq.org/download/
• Browser Specific– Chrome
https://sites.google.com/a/chromium.org/chromedriver/
– Safari http://selenium-release.storage.googleapis.com/2.46/SafariDriver.safariextz
Tester Workflow
57
• Selenium– Installationhttp
://geekpad.ca/blog/post/automating-browser-testing-with-phpunit-and-selenium• First you must download the Selenium server and
the Internet Explorer Server from http://docs.seleniumhq.org/download/
• Then you must download the Chrome server from http://chromedriver.storage.googleapis.com/index.html
Tester Workflow
58
• Selenium– Installation (continued)
• You do not need to download a Firefox driver since it’s builtin the main Selenium file.
• Once all files have been downloaded you must add your selenium folder to your existing path see http://www.java.com/en/download/help/path.xml
• Then you can launch the server using java. With something that looks like this– Mac - java -jar selenium-server-standalone-[version].jar
Tester Workflow
59
Why is Selenium cool?
•Records your actions
•Let’s you test expected outcomes
•Let’s you test them in a series
Tester Workflow
60
Example
•The developer was given a task to make sure that all instances of the word “Orange County” is replaced with the word “OC” on the website, for “legal reasons”.
•Demo
Jenkins
61
Jenkins
62
• Jenkins is an award-winning application that monitors executions of repeated jobs, such as building a software project or jobs run by cron. Among those things, current Jenkins focuses on the following two jobs:– Building/testing software projects continuously– Monitoring executions of externally-run jobs
https://wiki.jenkins-ci.org/display/JENKINS/Meet+Jenkins
Jenkins
63
Jenkins
64
Questions?
65
Resources
• http://www.dcs.bbk.ac.uk/~sven/cdm06/cdm3.pdf
• https://unit-tests.trac.wordpress.org/
• http://www.slideshare.net/ptahdunbar/automated-testing-in-wordpress-really
• http://johnregan3.co/2013/08/13/why-you-need-to-get-into-wordpress-unit-testing/
• http://blog.aizatto.com/2010/05/15/beginning-wordpress-automated-testing/
66
Resources
• https://www.owasp.org/index.php/Testing_Guide_Introduction
• https://make.wordpress.org/core/handbook/automated-testing/
• http://stackoverflow.com/questions/22414134/different-types-of-testing-eg-unit-functional-integration-etc-document
67
Resources
• http://www.seleniumhq.org/• http://geekpad.ca/blog/post/automating-browser-
testing-with-phpunit-and-selenium
• https://wiki.jenkins-ci.org/display/JENKINS/Meet+Jenkins
• https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins
68