Do not automate GUI testing
-
Upload
atila-inovecky -
Category
Technology
-
view
173 -
download
1
Transcript of Do not automate GUI testing
Do not automate GUI
Reinvention of wheel (HTTP performance testing) like an effective tool for regression testing
Atila Inovecký
Don't reinvent a wheel
API testing in 2001• Kaner, Bach, Pettichord: Lessons Learned in Software Testing – Lesson 132:
Automate tests using programming interfaces“…When we look at many product testing effort we’ve observed, we conclude that there is a strong correlation between the availability of programming interfaces for testing and the development of powerful automated test suites. Programming interfaces include APIs, command-line interfaces, COM interfaces, HTTP and more. …If you’re serious about automation, you’ll learn or get someone who knows this to help..…Focus your attention on where automation can help the most. Sometimes, you can effectively automate the GUI; other times you can’t. Don’t let people’s preconceptions of what automation has to look like limit you.”
Don't reinvent a wheel
Reasons for antipattern(traditional approach)• Automate everything!• Just QA is responsible for quality• Few testing rounds – no Continuous Integration• WHAT TO TEST(Business Functionality)=HOW TO TEST (GUI)?• “This small hotfix won’t break anything”• Production delivery 1 per year• DIDN’T FAIL YET• Trying to find a lot bugs
Don't reinvent a wheel
Test find bugs, not automation• Value of the regression test depend on value of the covered
functionality• Count of found bugs say nothig about value\effectivity of automated
test• Most of regression bugs is caused by unexpected impact of backend
integration not GUI• Automation is not medicine for wrong communication
Don't reinvent a wheel
Conclusions
• Unit testing is the core of regression testing• API testing is not the FUTURE but it is really just a reinvented wheel• Future of automated GUI testing is (Exploratory tests, …)???• GUI testing vs API testing
Don't reinvent a wheel
REST – Representational State Transfer• REST or RESTful web services are one way of providing
interoperability between computer systems on the internet. REST-compliant web services allow requesting systems to access and manipulate textual representations of web resources using a uniform and predefined set of stateless operations.
Don't reinvent a wheel
RESTful URLs• URLs intended to improve the usability and accessibility of a website
or web service by being immediately and intuitively meaningful to non-expert users.
Non-semantic URL Semantic URL
http://example.com/index.php?page=name http://example.com/name
http://example.com/index.php?page=consulting/marketing http://example.com/consulting/marketing
http://example.com/products?category=2&pid=25 http://example.com/products/2/25
http://example.com/cgi-bin/feed.cgi?feed=news&frm=rss http://example.com/news.rss
http://example.com/services/index.jsp?category=legal&id=patents http://example.com/services/legal/patents
Don't reinvent a wheel
Relationships between URL and HTTP methods
Uniform Resource Locator (URL) GET POST
Collection, such ashttp://mgs.com/resources/
List the URIs and perhaps other details of the collection's
members.
Create a new entry in the collection. The new entry's URI is
assigned automatically and is usually returned by the operation.
Element, such ashttp://mgs.com/resources/item17
Retrieve a representation of the addressed member of the collection, expressed in an
appropriate Internet media type.
Not generally used. Treat the addressed member as a collection in its own right and create a new
entry within it.
Don't reinvent a wheel
Important HTTP Request HeadersHeader field
name Description Example
Cache-Control Used to specify directives that must be obeyed by all caching mechanisms along the request-response chain. Cache-Control: no-cache
Content-Type The MIME type of the body of the request (used with POST and PUT requests).
Content-Type: application/x-www-form-urlencodedContent-Type: application/json
Content-Type: multipart/form-data
Cookie An HTTP cookie previously sent by the server with Set-Cookie. Cookie: $Version=1; Skin=new;
HostThe domain name of the server (for virtual hosting), and
the TCP port number on which the server is listening. The port number may be omitted if the port is the standard
port for the service requested.Host: qa16-ats.mgspdtesting.com
Don't reinvent a wheel
Important HTTP Status CodesStatus Code Description
200 Ok
Standard response for successful HTTP requests. The actual response will depend on the request method used. In a GET request, the response will contain an entity corresponding to the requested resource. In a POST request, the response will contain an entity describing or containing the result of the action.
302 Found Common way of performing URL redirection.
400 Bad Request The server cannot or will not process the request due to an apparent client error
403 Forbidden The request was a valid request, but the server is refusing to respond to it. The user might be logged in but does not have the necessary permissions for the resource.
404 Not Found
The request was a valid request, but the server is refusing to respond to it. The user might be logged in but does not have the necessary permissions for the resource. Usually means that the resource already didn’t exist.
501 Internal Server Error
Indicates that the request could not be processed because of conflict in the request, such as an edit conflict between multiple simultaneous updates.
Don't reinvent a wheel
Open Source Tools for API Testing• Performance testing tools• SoapUI (SoapUI NG Pro), Jmeter(SmartMeter, Blazemeter)
• Web Service testing tool• Postman, DHC, HTTPRequester, Wizdler, WebInject, Http Master, …
• Automation Frameworks• FitNesse, Cucumber, Robot Framework, …
Don't reinvent a wheel
SoapUI• Open-source web service testing application:• for service-oriented architectures (SOA) • for representational state transfers (REST).
• Covers:• web service inspection, invoking, development, simulation and mocking,• functional testing,• load and compliance testing.
• A commercial version, SoapUI Pro (Api Ready), which mainly focuses on features designed to enhance productivity.
Don't reinvent a wheel
SoapUI Open Source– pros and cons• Pros
• support HTTP(S), REST, SOAP, JDBC, AMF • IDE -Groovy -> command line, visible SoapUI API• User forums, tradition• Easy to learn, easy to use
• Cons• Team work support• No recording• Weak user documentation• Test history, test debugging, compare results• Buggy
Don't reinvent a wheel
Conclusions• Web application automated testing is possible through HTTP(S)• Web application automated testing is effective through REST• REST API popularity is growing
Don't reinvent a wheel
Fiddler• Captures HTTP and HTTPS traffic and logs it for the user to review
(the latter by implementing man-in-the-middle interception using self-signed certificates).• https://www.telerik.com/download/fiddler
Don't reinvent a wheel
HTTPS Traffic settings• Capture - F12 or Click on left bottom corner• Select captured process by• Preview - Tab Inspector• Request – Top part• Response – Bottom part
Don't reinvent a wheel
Fiddler - Inspector• Request Inspectors• Headers—Shows request headers and status.• Webforms – Shows parameters visualized as table from QueryString and Body• Raw—Shows the “real” request header and body
• Response Inspectors• Headers—Shows response headers and status.• SyntaxView—Shows the response body with HTML syntax• WebView – Shows the browser output• Raw – Shows “real” response header and body
Don't reinvent a wheel
Initial settings• File\Preferences – • Proxy Settings -> Manual 127.0.0.1:8888• UI Settings – Size of Req Response Message to Show – 1000000
• File\Import project (File\Create Empty Project)
Don't reinvent a wheel
What you get for 500$ in SoapUI NG Pro
• a• File\Import project (File\Create Empty Project)
Don't reinvent a wheel
Useful links
REST 101: The Beginner's Guide to Using and Testing RESTful APIs SoapUI - Getting Started with REST Testing SoapUI - Functional Testing SoapUI – Scripting & Properties SoapUI – API Docs
Don't reinvent a wheel
Reference List• Gojko Adzic, David Evans - Fifty Quick Ideas To Improve Your Tests• Mike Cohn - Succeeding with Agile: Software Development Using
Scrum• Lisa Crispin, Janet Gregory - Agile Testing: A Practical Guide for Testers
and Agile Teams• Dorothy Graham, Mark Fewster - Experiences of Test Automation:
Case Studies of Software Test Automation• Cem Kaner, James Bach, Bret Pettichord - Lessons Learned in Software
Testing: A Context-Driven Approach