Fujitsu Computer Systems Testing Considerations (Past experience, Methodology, Options…) CAMP F2F...
-
Upload
alfred-quinn -
Category
Documents
-
view
215 -
download
0
Transcript of Fujitsu Computer Systems Testing Considerations (Past experience, Methodology, Options…) CAMP F2F...
Fujitsu Computer Systems
Testing Considerations(Past experience, Methodology, Options…)
CAMP F2F Redwood City,November 2012
Jacques DurandFujitsu
1
Fujitsu Computer Systems
Testing serves different purposes
• Validating a specification
• Verifying an implementation
See: https://wiki.oasis-open.org/tab/TestingPolicy
2
Fujitsu Computer Systems
Testing serves different purposes
• Validating a specification• ambiguities, fuzziness• gaps (unspecified features)• inconsistencies / conflicts
• Verifying an implementation• conformance• interoperability: two+ implementations interoperable (or conforming artifacts portable?)• maintenance: is the implementation still conforming/interoperable over time? (changes in config, etc.)
See: https://wiki.oasis-open.org/tab/TestingPolicy
3
Fujitsu Computer SystemsFujitsu America, Inc.
TestCase
TestCase
Testassertion
Testassertion
SpecificationTest
assertions TestCase
is
addressing
ActualTest suite
Normativestatement
Normativestatement
Normativestatements
derivedfrom
Test assertions: a common test foundation
Test Assertion:• A “measurable/testable statement of behavior for an implementation”• Blueprint for an executable test unit
• see: https://wiki.oasis-open.org/tab/TestabilityGuide• see: https://www.oasis-open.org/committees/download.php/44696/taguidelines-v1.0-wd02.pdf
4
Fujitsu Computer SystemsFujitsu America, Inc.
Example: a specification on “widgets”
[requirement 1023] “A widget MUST be of cuboid shape”.
Here is a test assertion addressing this requirement:
TA id: widget-TA100-1 Target: widget Normative Source: “widget specification”, requirement 1023 Predicate: [the widget] has six facets, and each one of these is of rectangular shape. Prescription Level: mandatory
Worded:
• in knowledge of the testing ability
• understandable by a non-expert
5
Fujitsu Computer SystemsFujitsu America, Inc.
An example TA (CIMI)- The normative source is an interpretation (summary) of several normative statements in the original specification- The “SUT” tag identifies the larger entity “responsible” for conformance, while the “target” is a sequence of messages
6
Fujitsu Computer SystemsFujitsu America, Inc.
An example TA (SCA)- If the “prerequisite” is false, the “target” is not qualified for further evaluation of the “predicate”.- The tags here are just keywords used to classify this TA.
7
Fujitsu Computer Systems
What is to be tested
PaaSProvider
Black-box testing: through the testing of concrete test “targets” being exchanges: artifacts, messages
Major CAMP entities or roles
AppUser#1
Messages
Artifacts
8
Fujitsu Computer Systems
Conformance aspects to verify:
• (1) Message & Artifact well-formedness• Protocol aspects• schema validation, HTTP headers…• Request/Response basic consistency
• (2) Consistency between exchanges and what is currently deployed on the PaaS platform
• Right set of resources created, after operations?• message responses consistent with platform state?
9
Fujitsu Computer Systems
Benefits of Test Assertions (1):
• Improve specifications• by “translating” normative requirements in precise implementation behaviors, discover :
• gaps / ambiguities / inconsistencies
• Align implementations (before testing)• lead to a common interpretation of the spec.
• Facilitate Testing• starting point for Test Suites
10
Fujitsu Computer Systems
When to write Test Assertions ?
POC workTest SuitedeveloptProduct
dev
Implementationwork
Technical workOn Specification
Standardization(WDCS OS)
? ? ?
specificationwork
Best for maximum benefit:
while writing the spec
11
Fujitsu Computer Systems
A look at 3 Testing Set-ups: ( all useful )
PaaSProvider(SystemUnder Test)
TestDriver
- a single implementation of System Under Test (SUT) (CAMP) is being tested.- Primary goal: conformance assessment
1. Isolation Testing
12
Fujitsu Computer Systems
A look at 3 Testing Set-ups: ( all useful )
2. MatrixTesting
AppUser#1
- a test scenario is repeated for each pair of implementations(or pair PaaS Consumer-Provider)- main goal: interoperability assessment
AppUser#1
PaaSProvider A
AppUser#1AppUser#2
AppUser#1AppUser#3
PaaSProvider B
PaaSProvider C
13
Fujitsu Computer Systems
A look at 3 Testing Set-ups: ( all useful )
PaaSProvider(SystemUnder Test)
TestDriver
- concerns in-production systems- main goal: check for regressions- 2 variants: (a) driven tests (b) real exchange monitoring
3. Maintenance Testing
(b)
(a)
AppUser#1 In-production
Provider
14
Fujitsu Computer Systems 15
Separating Test Operation from Test Analysis• allows for reusing analysis test tool in various set-ups
Phase 1: Test operations
TestAnalyzer
Testassertions
Phase 2: Test analysis
ExecutionTrace
TestReport
Fujitsu America, Inc.
SUTTestDriver
SUT#1
SUT#2
SUT#3
users SUT
In-productionProvider
Isolation testing
Matrix testing
Maintenance testing
Fujitsu Computer Systems
Test Tools: overview
WebService
ANALYZER
MONITORInterceptor
Logger
Messageartifacts
Example of 2-phase testing: WS-Interoperability Testing
Execution report
Test report
Fujitsu America, Inc.
Phase 1
Phase 2
TestSCENARIOS
Client code
16
Fujitsu Computer Systems
Benefits of Test Assertions (2): when implementing test suites
• Single Test Tool/Suite for conformance logic• usable in all test set-ups
• Visibility of conformance logic• in test suite (blueprint for test cases)• in test report (each report item refers to a T.A.)
17