Magento Testing on all fronts
-
Upload
aoe -
Category
Technology
-
view
1.395 -
download
2
description
Transcript of Magento Testing on all fronts
On all Fronts Testing
Magento
San Francisco Magento Meetup Group - March 11, 2014
Fabrizio Branca Lead System Developer at
E-Commerce: Magento
CMS: TYPO3
Portals: ZF, FLOW,…
Mobile Searchperience: SOLR
120 people in 7 offices world-wide
High Performance
/Scale
Global Enterprise Projects
Unit
Service
UI Acceptance
http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid
What the customer thinks
the code should do
What the developer thinks
the code should do
Load Test
Continuous Integration
Deployment
Development
Delivery
Improvement
Quality Assurance Continuous
Everything sounds better with “continuous”
“Continuous” == “Doing stuff over and over again”
Builds Deployment Tests Infrastructure
Setup
Devbox Setup
== “Automation”
tim
e s
pe
nt
task size
does it manually
does it manually
gets annoyed
writes script to automate
runs script
wins
loses
Geeks vs. Non-Geeks
Makes fun of geek’s complicated method
— Ovidiu Platon
We are not shipping your machine! if it works on your machine! I don’t care
Picture The BIG
Vagrant Boxes
Awesome stuff
Every single line potentially
can break production!
Bratwurst (Germans)
Donut (Americans)
Pizza (Italians)
Version Control
GIT, SVN,…
PHP lint
Version Control Code Reviews
Gerrit, Crucible, Phabricator,…
Version Control Code Reviews Build
Files
DB
Installer
Settings
Triggered by commit, by time
or manually
Jenkins
minify js/css
.tar.gz
“virtual package” (revision)
Version Control Code Reviews Build
Files
DB
Installer
Settings
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
CodeSniffer
php-pmd
php-cpd
php-depend
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration”
First system where code from all
developers/teams meets each other
unstable system. Might be broken. Will be rebuild
without warning
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests
PHPUnit
Code coverage
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests
Test features that rely on other parts (e.g. framework, external
services,…) External components (SSL, Redis, Varnish,…)
Basically this is also PHPUnit, but
addressing other stuff
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests Acceptance Tests
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests Acceptance Tests
Install on
“Staging”
Behat (Cucumber)
Same setup as production, but different machines
Client QA / approval
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests Acceptance Tests
Install on
“Staging”
*http://www.slideshare.net/aoemedia/performance-measurement-and-tuning
Stress
Tests
Jmeter
Spin up EC2 instances in the cloud to produce
traffic*
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests Acceptance Tests
Install on
“Staging”
Stress
Tests
Install on
“Production”
Version Control Code Reviews Build
Files
DB
Installer
Settings
Static Code
Analysis
Install on
“Integration” Unit Tests Integration Tests Acceptance Tests
Install on
“Staging”
Stress
Tests
Install on
“Production” Enable Features
Version
Control Code
Reviews Build Static Code
Analysis
Install on
“Integration”
Unit Tests Integration
Tests
Acceptance
Tests
Install on
“Staging”
Install on
“Production” Enable
Features
with EcomDev_PHPUnit Unit Tests
https://github.com/EcomDev/EcomDev_PHPUnit
use the “dev”
branch!
Config Model Block Helper Controller
fat model, skinny controller…
Classes PHPUnit_Framework_TestCase EcomDev_PHPUnit_Test_Case
EcomDev_PHPUnit_Test_Case_Config EcomDev_PHPUnit_Test_Case_Controller
Assertions
Fixtures
Mocks
Data Provider
Magento Integration
Testsuite management
Framework integration
Create test environment (db+config)
Manual
http://www.ecomdev.org/2011/05/24/ecomdev_phpunit-manual-version-0-2-0-is-available-for-
download.html
Video tutorial
http://www.ecomdev.org/2012/01/31/test-driven-magento-development-seminar-video-and-
code.html
Documentation More Information
With Selenium Acceptance Testing
Acceptance Tests
What’s out there? Selenium for PHP
PHPUnit_Selenium (Sebastian Bergmann)
WebDriver-PHP (Chibimagic)
Php-webdriver-bindings (Lukasz Kolczynski)
php-webdriver
PHPUnit_Selenium (Giorgio Sironi)
php-webdriver Selenium 2
08/2011 Facebook/php-webdriver (Justin Bishop)
Element-34/php-webdriver (Adam Goucher)
instaclick/php-webdriver (Anthon Pang)
php-webdriver
“A very thin wrapper of WebDriver”
So what?
A thin layer good! But not enough…
no test logic
no higher level components
dealing with Selenium specific tasks
not compatible to legacy test cases
No abstraction / structure
Architecture
HTTP AUT
No direct access (DB, classes,…)
REST* (over HTTP)
*http://code.google.com/p/selenium/wiki/JsonWireProtocol
Project test cases
PHPUnit
Project Components
Components Library
Menta
php-webdriver
reusable
project specific
Menta Selenium 2 Testing Framework for PHP
Who is “Menta”?
What is “Menta”?
Features Menta The framework.
Not the cat…
Components Session Management
? Bonus Features :) Event / Observer Configuration Management
PHPUnit integration
Why?
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
Data Providers
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
Dependent Tests
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
✔ ✔ ✔
Test Infrastructure
Menta for PHPUnit Abstract test class
PHPUnit_Framework_TestCase Menta_PHPUnit_Testcase_Selenium2 ProjectY_Abstract_TestCase
Very thin layer on top of PHPUnit
ProjectY_TestCase_Registration ProjectY_TestCase_Checkout the “magic” happens in the components
HTML Report Screenshots (on demand
and on failure)
test description from phpdoc
additional messages
error details
HTML Report “Duration Heatmap”
shows long running tests
Menta for PHPUnit Text Result Printer
PHPUnit 3.7.20 by Sebastian Bergmann. Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml > SUITE: > SUITE: > SUITE: Acceptance_Tests_Account_Register > TEST: Acceptance_Tests_Account_Register::register SUCCESS. (Duration: 8.18 sec) > TEST: Acceptance_Tests_Account_Register::login SUCCESS. (Duration: 6.5 sec) > TEST: Acceptance_Tests_Account_Register::testRegistrationMail FAILURE: Searching for mail with subject 'Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae' timed out (Duration: 22 sec) < Duration: 36 sec < Duration: 36 sec > SUITE: > SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest > TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCart FAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat(' ', @class, ' '), ' attentionText ')]" does not contain text "Bulb Vase" Failed asserting < Duration: 9.87 sec > SUITE: Acceptance_Tests_Checkout_CheckoutTest > TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryView SUCCESS. (Duration: 1.79 sec) > TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart FAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::checkout" to pass. (Duration: 5.34 sec)
Components
Common Assert Wait Screenshot Selenium1Facade
Imap WebService W3C Validator Curl …
Some might not even require a Selenium session
Components! PageObjects?
PageObjects Components
{ } }⊇{
PageObjects are
components
PageObjects?
represents services offered by a page or feature
is the only thing with a deep knowledge of
the HTML
clickCheckoutButton() isInCart($product) getTotals()
✔ ✔ ✔
Component Libraries
Menta
Magento
(your library)
Project-
specific
Magento
(community-provided)
General
(your library)
Gallery Screenshot
Perceptual diffs Build X
Build X-1
Perceptual Diff
github.com / AOEmedia / Menta github.com / AOEmedia / Menta_SampleProject
It’s composer based and you’ can get started in a few minutes!
With JMeter Load Testing
JMeter
jmeter.apache.org Current version:
2.11
JMeter HTTP
Simple Setup
JMeter
Slave
JMeter
Slave
JMeter
Slave
JMeter
Master
Master/Slave Setup
JMeter
Slave
JMeter
Slave
JMeter
Slave
JMeter
Master
EC2 Setup
JMeter
Slave
JMeter
Slave
…
Creating JMeter tests
JMeter
Test
Manually
Replaying Apache Logs
Recording Browser
interaction using a proxy “Convert” existing
acceptance tests
CSV Files
(Selenium)
HTTP
HTTP Behat Selenium Firefox Proxy
JMeter
Test
Converting Behat to JMeter
Using a Proxy JMeter
Using a Proxy Browser configuration
Manual configuration
Browser Extension
Using a Proxy Browser configuration
java -jar selenium-server-standalone-2.40.0.jar
-trustAllSSLCertificates
-Dwebdriver.firefox.profile=jmeter
Using a Proxy Selenium configuration
http://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.pdf
https://wiki.apache.org/jmeter/TestRecording210
http://jmeter.apache.org/usermanual/component_reference.html#HTTP%28S%29_Test_Script
_Recorder
Using a Proxy
https://jmeter.apache.org/usermanual/test_plan.html
• ThreadGroups
• Controllers
• Samplers
• Logic Controllers
• Test Fragments
• Timers
• Assertions
• Configuration Elements
• Pre-Processor Elements
• Post-Processor Elements
Elements of a Test Plan
http://www.slideshare.net/shubelal/abj-meterbrief-upladed/7
Elements of a Test Plan
• Tree structure
• Inheritance (Cookie Manager, Header Manager,…)
• Timers
• Variables
• Assertions
• Random values
• Scripting
• Extracting values
• Following redirects
• (check examples in bin/examples)
Basic Concepts
Workflows Search / Filters Static Pages
404
What to test Consider conversion rate /
funnel!
JMeter Plugins
jmeter-plugins.org
Distributed load testing
General Master/Slave setup
https://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.pdf
AOE presentations
http://www.slideshare.net/axeljung01/jmeter-and-amazon-cloud-workshop-t3dd2012
http://www.slideshare.net/aoemedia/performance-measurement-and-tuning
On EC2
https://github.com/oliverlloyd/jmeter-ec2
http://www.http503.com/2012/run-jmeter-on-amazon-ec2-cloud/
Server Monitoring
Thank you! Any questions?
http://www.aoe.com
http://www.fabrizio-branca.de
@fbrnc Follow me on twitter!
My blog