Web App Testing - A Practical Approach
-
Upload
walter-mamed -
Category
Documents
-
view
12.591 -
download
10
description
Transcript of Web App Testing - A Practical Approach
![Page 1: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/1.jpg)
11:00 AM – 12:00 PM
INSTRUCTOR: WALTER MAMED COMPANY: JWT.COM
Testing Web Applications: A Practical Approach
Friday, April 23rd
2010
![Page 2: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/2.jpg)
This page intentionally left blank.
![Page 3: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/3.jpg)
1
Testing Web Sites & Applications
A Practical Approach
Walt Mamed
JWT, Director, Quality Assurance
02/09/2010
News Story ‐ Stray Mouse Click
2
![Page 4: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/4.jpg)
2
Double Click of Death
• On November 14, 2007 at 3:30pm one of Credit Suisse’s trading algorithms suddenly went haywireSuisse s trading algorithms suddenly went haywire
• Sent hundreds of thousands of bogus requests to the exchange.
• Acted like a denial‐of‐service attack on the NYSE
• Affected trading of 975 stocks
• Caused by a trader who accidentally double‐clicked an icon in a trading program’s interface.
• Credit Suisse assessed a $150,000 fine
3
Overview
1. Test Planning
2. Configuration Management
3. Test Execution
4. Projects – CBT, iPhone, Security, and more
5. Test Tools
(Hint: Watch for gold nuggets)
4
![Page 5: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/5.jpg)
3
Test planning for a new or existing web site
TEST PLANNING
5
Test Planning
Test Planning is one of the keys to project success:
G th i R i tGathering Requirements
Functional Decomposition
Risk Based Testing Analysis
Develop Test Plans and Procedures
6
![Page 6: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/6.jpg)
4
Requirements Gathering
Requirements come in many forms:
P j t Pl B i R i t SProject Plan or Business Requirements Spec
Software Requirements Specification
Functional Design Document
Feature Specification Document
Interface Control Document
Use Cases
Wireframes
7
Functional Decomposition
FD ‐ Breaking it down piece by piece:
D th i t d d f ti i t b f tiDecompose the intended function into sub‐functions
Divide and conquer (split due to volume of effort)
Top‐down: if system is fully described
FD based on the flow of data or traversal by user
Verify all requirements have been covered.y q
It’s easier if you have Use Cases
More challenging if you have Business Requirements
8
![Page 7: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/7.jpg)
5
Functional Decomposition
Booking
Balancing risk basedtesting and repetition f t k
Critical or High use ofsubsystem, functionor feature
Smoke Tests
Booking
BookingScenarios
of tasks
Defect
or feature
Searches &Filters
Functional 1
Loyalty Program
DefectEscapesBusiness
Adjustments
Tools supportingmanual testing
Functional 1
Functional …
Functional …N
9
Test Planning – RBT Analysis
RBT ‐ Risk Based Testing analysis:
Hi h f b t f ti f tHigh use of a subsystem, function or feature.
Criticality of a subsystem, function or feature, including the cost of failure.
Prioritize what should be tested first.
Not doing so explains why big bugs are found at the end of a test cycle; its human nature to test the easy functionality first.
10
![Page 8: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/8.jpg)
6
Test Planning – RBT Analysis
Test Design Techniques using software models:
E i l titi iEquivalence partitioning− Breakdown elements into classes
− Perhaps use a mind map
Boundary value analysis− Identify edges or end‐points
Decision tables
State transition diagrams− Will also help define your negative tests
11
Test Planning ‐Mindmap
12
![Page 9: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/9.jpg)
7
Test Planning – Test Plans
Develop Test Plans and Procedures:Test plans are usually in WordTest plans are usually in Word.Detailed test procedures are usually in Excel.Quality Center – Requirements and Test Plan modules (great for confirming all requirements covered).Writing these documents should be easier if theWriting these documents should be easier if the previous steps (FD & RBT) were performed.IEEE 829 defines many types of test specifications– “If it’s not written down, it didn’t happen.”
13
Test Planning
Practical Suggestions for Test Planning :
W b A l ti (f i ti b it )Web Analytics (for existing websites)
Planning Test Automation? – Testability as a requirement for Development
Production Monitoring
14
![Page 10: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/10.jpg)
8
Test Planning Suggestions
Web Analytics (for existing websites)
O li B i O ti i ti (T l f O it )Online Business Optimization (Tealeaf, Omniture)– Exit rate, average time on page, contribution to revenue.
– Where and why are visitors leaving.
– Know how customers are using your site.
Browser usage (Cross Browser Test planning)d h b l– Browser type used, what version, mobile user type
– Use what you know your customers/visitors use.
Behavior Map (page hit frequency)
15
Test Planning ‐ Behavior Map
16
![Page 11: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/11.jpg)
9
Test Planning ‐ Testability
Planning Test Automation?
T t bilit i t f D l tTestability as a requirement for Development:
Provide a unique and meaningful name property for:– Every actionable html object on the page. (entry‐fields, buttons, radio buttons, dropdown list boxes, images, links, etc.)
– Every table object that requires testingEvery table object that requires testing.
– Every response that requires testing. The responsemay be in tables, spans, divs, lis, etc.
17
Test Planning ‐ Testability
Testability as a requirement for Development:
P l t th ‘id’ d ‘ lt’ t t i QAPopulate the ‘id’ and ‘alt’ tags to give QA more alternatives to identify an object during scripting.– SEO and 508 Compliance contribute to this recommendation as well.
Use a naming convention that includes the function or purpose of the given objector purpose of the given object.
Do not change any HTML element property name (including id & alt tags) from release to release.
18
![Page 12: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/12.jpg)
10
Test Planning ‐Monitoring
Production Monitoring:Ensure your site and applications are performingEnsure your site and applications are performing.Identify, resolve and prevent issues.Develop an escalation policy, triage, remediate, and confirm resolution.Use automated daily smoke tests to supplement monitoring from a customer or partner perspective.monitoring from a customer or partner perspective.Discuss this during the requirements phase– What, how, where and who?– The wrong time is the day of deployment.
Make sure you know what you are testing.
CONFIGURATION MANAGEMENT
20
![Page 13: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/13.jpg)
11
Configuration Management
Manage software configurations:
A dit fi ti ft h t QA/P dAudit configuration after push to QA/Prod– Use mySite.com/revision.txt to confirm
– Output contains Build Version, Date & Time
Establish method to directly access web servers– Avoid round‐robin approach behind load balancers.
h // b /– http://web#‐www.mySite.com/revision.txt
21
Let’s get to it!
TEST EXECUTION
22
![Page 14: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/14.jpg)
12
Functional Testing
Functional testing:
R S k S it C iti l P th t tRun Smoke, Sanity, Critical Path tests
Check all links and web pages– Site spider
• Start at Home page and traverse whole site– (exclude external links)
• Check HTTP status 2xx, 3xx, 4xx & 5xx
• View pages for gross or cosmetic failures. (more later)
– Xenu link sleuth
23
Functional Testing
Functional testing:
F b itt lForms submittal
User profiles
Role based access
Flash, Ajax, j
Back office testing
Examine server side logs
24
![Page 15: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/15.jpg)
13
Examine Server Side Logs
25
Usability Inspection
Usability Inspection:
N i tiNavigation
Page Content
Intuitive
508 Compliance ‐ accessibility
Search
Sitemap
Help
26
![Page 16: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/16.jpg)
14
Usability – User Experience
Page Download Times and Browser Rendering:
N lik l b itNo one likes a slow website
Load testing and performance usually done late in the test cycle
Measure web page download performance early– Part of Sanity/Smoke test script.
– Run multiple times and average.
Track page download trends from release to release.
– Test script writes download times to csv.
27
SEO Dated Page Download Trend XPIE6 (Single User)
20
25
38b5
R6
42b2
10
15
44b4
47b4
49b2
0
5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
28
![Page 17: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/17.jpg)
15
Page Download and Render Time
29
Usability – Drilling Down
Page Download Time and HTML elements:
M HTML l t d l d tiMeasure HTML element download times– HttpWatch (works with both Firefox & IE, has an API)
– tools.pingdom.com (to demo object downloads)
– Yslow (Firefox addon)
30
![Page 18: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/18.jpg)
16
HttpWatch
31
Cross Browser Testing, iPhone automation, Security, Defect Life Cycle
TESTING PROJECTS
32
![Page 19: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/19.jpg)
17
Cross Browser Testing
Cross Browser Testing project:
C t d d d CBT l bCreated and used a CBT lab.– Various combinations of FF, IE and Windows OS
• XP/IE6, XP/IE7
• Vista/IE7
• XP/Firefox
– Ran automated regression tests on each combination.
– Discovered many cosmetic defects.
– No functional errors found.
– Many companies use Selenium.
33
Cross Browser Testing
Cross Browser Testing project (next steps?):
C id i HTML/CSS t h k / W3CConsidering an HTML/CSS syntax checker / W3C validator– Many online tools generate considerable output.
– HTML Validator (Firefox Add on)
– Total Validator (Firefox Add on)Litmusapp com is another consideration– Litmusapp.com is another consideration
34
![Page 20: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/20.jpg)
18
Automated testing on the iPhone
Automated testing on the iPhone:
A H t l b it t d f th iPhA Hotels.com website was created for the iPhone
Examined test tools to automate testing of iPhone web site.
Tried SafariWatir on the Mac without success.
As a reasonable alternative I used FireWatir (Watir for Firefox) on a PC and ran automated regression tests for the iPhone web site.– Minor visual differences vs. Safari
35
Security Testing
Security – start simply (perhaps you already do?):
I lid i t i t t t fi ld d fInvalid inputs in text entry fields and forms
SSL– https is used where appropriate (e.g. forms)
Internal URLs not accessible (unless logged in)
Confirm no access to web server directories
XSS – Cross Site Scriptingp g
36
![Page 21: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/21.jpg)
19
Security Testing
Security:
S t ' t tiSet everyone's expectations
Gather good tools
Look at your application from every perspective
Test for underlying weaknesses
Go back and verify your scanner findingsy y g
Manually check for weaknesses
Test your source code
37
Security Testing
Security – Captcha:
C t h d t t t t d i t i th ild dCaptcha – detects automated scripts in the wild and blocks them– Verifying the detection of automated scripts is easy if you have automated tests
– In order to run automated tests in Production, plan to have a means to disarm Captchap
– Timed re‐arming is preferable such that Captcha is enabled automatically to protect the site in case you forget.
38
![Page 22: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/22.jpg)
20
Defect Lifecycle
Managing the defect lifecycle:
I d t il d d i ti d iIssues are detailed, descriptive, and concise.
Ensure severity and priority are appropriate.
Ensure there are no unassigned issues.
Hold weekly mandatory review meetings between QA & stakeholders
Write a defect, write a test case (if none exists)– Copy steps to reproduce into a new test case. ^C^V
– Great way to “beef up” regression test suite.
39
What’s in your QA Tool Belt?
TEST TOOLS
40
![Page 23: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/23.jpg)
21
Test Tools – Browser Add‐ons
Useful Firefox Add ons:
Fi bFirebug
FormSaver
FireCookie
tamperData (view/modify HTTP/HTTPS)
Screengrabg
Xpather
41
Test Tools – Browser Add‐ons
Useful Internet Explorer Add ons:
D l T lbDeveloper Toolbar
IECookiesView
Fiddler (Watcher – Passive Security Auditor)
Webcollect (screen capture)
Web Accessibility Toolbary
Mathon (Swiss army knife)
42
![Page 24: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/24.jpg)
22
Test Tools ‐WATiR
Automated Test Tool:
WATiR W b A T ti i R bWATiR – Web App Testing in Ruby– Supports your web app no matter what it is developed in
– Full featured modern scripting language
– Supports multiple browsers on different platforms
– It is powerful and easy to use, yet beautifully lightweight
There is an active and growing community behind it– There is an active and growing community behind it
– It is free Open Source tool. There are no costs to use the tool
– User for five years (solid, stable, growing functionality)
43
Test Tools ‐WATiR
Automated testing results using Watir:
F ll t t d th H t l t tiFully automated the Hotels.com testing– Sanity testing (page download times too)
– Regression Testing (a deployment every week)
– Booking tests were data driven (Excel spreadsheet)
– Three day test cycle (2 resources) reduced to two hours end to end Exploratory testing added to processend to end. Exploratory testing added to process.
– Data Center Consolidation (15 app servers, 8 instances on each, 120 total instances)
44
![Page 25: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/25.jpg)
23
Test Tools ‐ Justification
Tips when using automation:
U id it i t it d tUse a widescreen monitor in portrait mode to maximize visibility of the whole page.
Use automation to scroll to the bottom of the page.
Record all defect #’s detected by automated testing– Application Services (weekly releases)
– Projects (web page redesigns)
Branch test scripts to mimic Development code
Calculate your ROI
45
Test Tools – Software Utilities
Test Utilities:
R b i f ll f t d i lRuby is a fully featured programming language.– Gems like NET::SMTP to send mail or pop mail
• Verify emails sent from web app; like change password, click link.
– NET::SSH tail utility pulls server logs to desktop for viewing
– Missed destinations utility (feedback to Dev)
– Run SEO tests on web pages (too tedious to do manually)Run SEO tests on web pages (too tedious to do manually)
– Site Spider that traverses site starting at the home page• Able to traverse the whole site with minimal scripting time.
46
![Page 26: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/26.jpg)
24
Test Tools ‐ Security
Security:
OWASP b it t ti t lOWASP.org – web security testing tools
Ethical Hacker Network
HP Dev Inspect (for programmers)
HP QA Inspect (for QA testing)
HP Web Inspect (for Production)p ( )
Hosted services; McAfee for production security testing.
47
Test Tools ‐ Performance
Social Networking:FacebookFacebook– Can’t “load test” in Facebook’s domain.– Created simulateUser.php (randomized actions)
• Register new friends• View canvas, tag other friends with characteristics, save profile
– Ran apachebench (ab) against simulateUser.phpFacebook application refactored in targeted areas– Facebook application refactored in targeted areas
– Placed database in RAM (limited risk)– 312 to 46K requests/transactions (150 X better performance )
![Page 27: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/27.jpg)
25
Test Tools ‐ Flash
Flash Testing Apps ‐ Commercial:QTP with plugin (instrument the Flash code)QTP with plugin (instrument the Flash code)TestCompleteRanorexAutoCzarTestPlant ‐ EggPlant (image based)
Fl h T ti A O SFlash Testing Apps – OpenSource:T‐Plan Robot ASUnit
49
Quality Nuggets
Before Deployment Day!Run your regression test scripts in Prod why?Run your regression test scripts in Prod – why?– Deployment failed, troubleshooting focused on new release, root cause was a pre‐existing condition in Prod.
– Discovering issues before deployment eliminates the confusion and unnecessary troubleshooting from assuming that a new deployment caused the problem.
– By running automated regression the evening before aBy running automated regression the evening before a deployment, several issues have been found since, some serious.
– Content Management System changes
50
![Page 28: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/28.jpg)
26
QUESTIONS?
51
Biography – Walter Mamed
Walter Mamed is Director of Quality Assurance at JWT (Digital Technology) in Irving, Texas. He has over 30 years experience in a variety of quality assurance and software test engineering development positions, focusing on software and hardware test automation.
Walt has been building test automation frameworks for GUI testing and web based applications for over 15 years. His web testing experience includes secure Email, On‐boarding, ecommerce and lead generation as well as large‐scale automated regression test suites.
Walt is very active in the professional community as Director of the Board /and Secretary for the Dallas/Ft. Worth (HP) Mercury User Group
(DFWMUG.com) for the last 7 years. He is an ASQ Certified Software Quality Engineer.
52
![Page 29: Web App Testing - A Practical Approach](https://reader033.fdocuments.us/reader033/viewer/2022061208/5485ec7b5806b59a588b4864/html5/thumbnails/29.jpg)
27
Acronyms
• API – Application Program Interface
• CBT – Cross Browser Testingg
• CMS – Content Management System
• CSS – Cascading Style Sheets
• CVS – Concurrent Versioning System
• ETL – Extraction, Transformation, and Loading
• FD – Functional Decomposition
• RBT – Risk Based Testing
• RCS – Revision Control System
• SVN – SubVersion
• SQL – Structured Query Statements
• W3C – World Wide Web Consortium
53