An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing
description
Transcript of An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing
1
An Exploration of Errors in Web-based Applications in the Contextof Web-based Application Testing
PhD Proposal
Kinga Dobolyi
May 2009
2
The shopping cart
3
The shopping cart
4
The shopping cart
5
What is going on
• Problem: faults in web-based applications cause losses of revenue, and they are hard to test
• Approach: study errors in web-based applications in the context of web testing
• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing
6
Outline
• Introduction and motivation
• Thesis statement
• Background
• Goals and approaches
• Preliminary work
• Expected contributions
7
Motivation
• Testing of web-based applications in particular deserves further examination due to economic considerations:– Monetary throughput: Backbone of e-
commerce and communications businesses– Customers: low customer loyalty– Development: Companies are choosing not to
test due to resource constraints
8
Motivation: e-commerce
• Internet usage: 73% of people in the US in 2008– Browsers are dominant application– $204 billion in Internet retail sales annually
• Global online B2B transactions total several $trillions annually
• One hour of downtime at Amazon.com cost $1.5 million dollars
• 70% of major online sites exhibit user-visible failures
9
Motivation: customers
• Customer loyalty is notoriously low– Determined by the usability of the application
[Offutt 2002]– Freedom and options
10
Motivation: customers
• Lesson learned: web-based applications need to be well-designed and well tested
• Are they?
11
Motivation: development
• Technology
challenges:– Heterogeneous, opaque
components – Dynamic page content generation– Persistent state operated upon by
concurrent, global users around the clock
12
Motivation: development
• Web-based applications are often not tested– Enormous pressure to change
• Short delivery times, high developer turnover rates, and quickly evolving user needs
– No formal process model
13
Motivation: summary
• Problem: faults in web-based applications cause losses of revenue, and they are hard to test
• Approach: study errors in web-based applications in the context of web application testing
• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing
14
Thesis statement
• Hypothesis: web-based applications have special properties that can be exploited to build tools and models that improve the current state of web application testing and development:– Tend to fail in predictable and similar ways– Human centric definition of acceptability
15
Thesis statement
• Problem: faults in web-based applications cause losses of revenue, and they are hard to test
• Approach: study errors in web-based applications in the context of web testing
• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing
16
Background: testing techniques
• Non-functional (static) validation– Server load testing– Link testing– HTML/spelling validation
• Modeling approaches
• Capture-replay– User session-based testing
17
Background: oracles
• Oracles (oracle-comparator)
1 < <P>The same table could be indented.
2 < <TABLE border="1">
3 ---
4 > <p>The same table could be indented.</p>
5 > <table border="1" summary="">
– False positives from
diff-like tools– Want precise comparators
18
Background: automation
• Automation– Test case generation: VeriWeb, PHP– Test case replay
• URL + post-data
– Failure detection
19
Background: metrics
• How do we measure success?– Code coverage– Fault seeding
• Human• Automatic
– Cost
• How do we know these are indicative of the real world?
20
Background: fault definition• Defining an error:
– “the inability to obtain and deliver information, such as documents or computational results, requested by webusers.” [Ma & Tian 2003]
• Fault taxonomies– Figure from
[Marchetto et al2007]
21
Background: challenges
• Functional validation remains a challenge– Regression testing should be more precise
and automatic
• We do not know if test suite efficacy metrics are indicative of the real world– We should examine the severity of uncovered
faults
22
Goals and approaches
• Problem: faults in web-based applications cause losses of revenue, and they are hard to test
• Approach: study errors in web-based applications in the context of web testing
• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing
23
Goals and approaches
• I propose to:– Model errors in web-based applications
• Identify them more accurately • Automate the oracle-comparator process
– Make web testing more cost-effective• Devise a model of fault severity that will guide test
case design, selection, and prioritization• Validate or refute the current underlying
assumption that all faults are equally severe in fault-based testing
24
Goals and approaches: Goals
• Reduce the cost of regression testing web-based applications – Use special structure of web-based application output
to precisely identify errors
• Automate web-based application regression testing– Unrelated web-based applications tend to fail in
similar ways
• Understand customer-perceived severities of web application errors.
25
Goals and approaches: Goals
• Formally ground the current state of industrial practice– Validate or refute fault injection as a standard for
measuring web application test suite quality
• Understand how to avoid high-severity faults during web application design and development
• Reduce the cost of regression testing web applications by exposing high-severity faults– Test case design, selection, and prioritization (test
suite reduction)
26
Goals and approaches: Outline
27
Goals and approaches: Step 1 – oracle-comparator
• Construct a precise oracle-comparator that uses the tree-structured nature of XML/HTML output and other features– Model errors on a per-project basis– Semantic distance metric to reduce false
positives
28
Goals and approaches: Step 2 – automated oracle-comparator
• Exploit the similar way in which web applications fail to avoid the need for human annotations in training a precise oracle-comparator– Train a precise oracle-comparator on data
from other, unrelated web applications– Use fault injection to improve the results when
necessary
29
Goals and approaches: Step 3 – human study
• Conduct a human study of real-world fault severity to identify a model of fault severity– Severities different than self-reported in bug
repositories– Screenshots of current-next idiom– Also survey developers
30
Goals and approaches: Step 4 – fault seeding validation
• Compare the severities of real-world faults to seeded faults using human data (validate fault seeding)– The severities of seeded errors have uniform
distributions?– The severity distribution of seeded errors
matches the distribution of real-world errors, according to the results of the survey from Step 3?
31
Goals and approaches: Step 5 – software engineering guidelines
• Identify underlying technologies and methodologies that correlate with high-severity faults– As an alternative to testing– Tie high severity errors to underlying code,
components, programming languages, and software engineering practices
32
Goals and approaches: Step 6 – testing techniques
• Identify testing techniques to maximize return on investment by targeting high-severity faults– Introduce a new metric for the (web
application) test suite reduction research community
33
Preliminary Work: Step 1
• Step 1: Construct a precise oracle-comparator using tree structured XML/HTML output and other features– Multiple versions of
10 open-source
benchmarks– 7154 pairs of oracle-
testcase output, 919
of which labeled as
“possibly an error”
34
Preliminary Work: Step 1
• Evaluation: F-measure (precision and recall) using our model
35
Preliminary Work: Step 1• Longitudinal study to measure effort saved
– Calculate cost of looking : cost of missing– Low ratio means we are saving effort
36
Preliminary Work: Step 2
• Step 2: Exploit similarities in web application failures to avoid human annotations when training a precise oracle-comparator– Same setup as Step 1– Use existing, annotated pairs of test-oracle
output from unrelated applications to train a comparator for the application at test
37
Preliminary Work: Step 2
• Evaluation: measure precision and recall
38
Preliminary Work: Step 2
• Use fault seeding to introduce project-specific faults into the training data set
39
Preliminary Work: Step 3
• Step 3: Model real-world fault severity based on a human study.– Collect 400 real-world faults
• Evaluation:
have subjects
use the model
to classify
faults
40
Preliminary Work: Step 4
• Step 4: Compare the severities of real-world faults to seeded faults using human data.– Test same human subjects with 200 + 200
manually-injected and automatically-injected faults
– Conduct a survey
of developers for
fault severity
41
Preliminary Work: Step 5
• Step 5: Identify technologies and methodologies that correlate with high-severity faults
• Do high severities correlate with:– Programming Language– Level in three-tier architecture– COTS component– User error (usability issue)– Type of error (business logic, resource allocation, syntax error,
etc)– Fault taxonomies (existing)– Surface features of visible output: white screens, stack traces,
misspellings, formatting errors
• Evaluation: developer survey (time permitting)
42
Preliminary Work: Step 6
• Step 6: Identify testing techniques to target high-severity faults– Targets testing– Assign a testcase a severity rating a priori
• Evaluation: compare the severity of faults exposed with our technique versus other test suite reduction approaches
43
Expected Contributions
• Problem: faults in web-based applications cause losses of revenue, and they are hard to test
• Approach: study errors in web-based applications in the context of web testing
• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing
44
Expected Contributions
• Reduce the cost of regression testing by constructing a precise oracle-comparator
• Develop a model of customer-perceived severities of web application faults
• Validate or refute fault injection as a standard for measuring web application test suite quality
• Propose new software engineering guidelines for web application development and testing
45
Questions?
46
Original Contributions
• Fault Severity Model– Severity has not been studied in this domain, and
customers are an integral part of these applications– Providing a new metric to the research community– Validate/refute fault seeding
• Precise-oracle comparators– First to use different versions of benchmarks– Can be completely automated– XML and HTML
47
Expected Impact
• Fault Severity Model– Can be applied to testing techniques in this field to
make them financially feasible for developers– Change the way in which test suite efficacy is
measured– Potentially impact web site design as usability issues
may become more evident• Precise oracle-comparator
– Automation makes it much more feasible for adoption than existing techniques
– Potentially allow companies to conduct regression testing if they were not testing beforehand
48
Timeline
• Steps 1 and 2: precise comparators– Completed, 2 papers under submission
• Steps 3 and 4: human study– Data collection completed, analysis under way for submission of
1 paper– Expected completion by September
• Step 5: software engineering guidelines– Expected completion by October– Expected 0.5 paper
• Step 6: testing according to fault model– Expected completion by February– Expected 1 paper