Interoperability Testing

Work done so far

WSDL subgroup Generated Web Service Description with

aim for maximum interoperability between various SOAP stacks

WSDL tested against tooling supplied by JAX-RPC RI AXIS .NET

Basic and informal interop tests .NET <-> AXIS

WSDL subgroup (cont’d)

Released a subgroup report (Andre) Philosophy followed

e.g. choosing document/literal style Extension mechanism Found problems/concerns Raised bug reports against the stacks

AXIS begun work on all of them .NET reported fixes were in progress JAX-RPC RI has not responded yet

Fixes/workarounds reflected in the WSDL and interfaces

Interoperabilty

WSDL subgroup work Allows various stacks to generate

code from WSDL definitions Ensures interoperability on SOAP level

Need for interoperability on WSRP protocol level cross-vendor testing Compliance testing ?

Cross-vendor testing

Release open source project which provides sample implementation

Allows vendors to test their Producers and Consumers against this implementation

Does not guerantee complianceVendors willing to make their Producers accessible to others for interop tests?

Allows others to test their Consumers In turn vendors return feedback on their Producer

implementations Requires similar (spec) level of implementation

Should be easy once version 1 is released

Still no compliance testing in a formal sense

Cross-vendor Testing - Matrix

Consumer/Producer

Vendor1

Vendor2

Vendor3

Vendor4

Vendor1

Vendor2

Vendor3

Vendor4

Compliance Testing

Testing a particular implementation for compliance with the specification.Compliance testing is strictly “black box” testing

Compliance Testing - Tasks

Develop a set of testable assertions which represent the specification

Derive testable assertions from the specification Determine test cases implied by each assertion

Develop a Compliance Test Suite available for public download so that vendors and customers can verify that their implementations are compliant with the WSRP 1.0 Version of the spec

Design, develop, and test the tests for each assertion Create a test framework to run, evaluate, and report

on the tests

Deriving Testable Assertions

Significant Effort to recast the spec as assertionsRequires committee involvement

Test Suite Framework

A technical framework for running compliance tests.The framework shall be used to run, evaluate, and report test cases.Should be highly scriptable and configurable so test cases may be added easilyThis may be a significant effort by itself, depending on our ability to leverage existing frameworks

Testing Producers

Develop a Consumer which executes test cases against a Producer and reports the results.Test cases are configured as XML inputReports are generated as XML output

Testing Consumers

Develop a RI Producer conforms to the spec (I.e. passes all

the compliance tests) produces reports on the behavior of

the Consumer, if in error.

For example, validating the state that was supposed to be kept and returned by the Consumer

Testing - Roadmap

Form interop test subgroup Start interop tests February, 15th? Establish interoperability April, 15th

Form compliance test subgroup Draft assertion suite March, 15th Finalize assertion suite May, 1st

Interop

Mike AlanRichardAndreNigelSubbuGil(?)

Compliance

RossDanRich

Backup

Existing Examples

Java Community Process Calls for developing a Technology Compatibility Kit

(TCK) side by side with Reference Implementation Well developed process and tools

W3C The W3C standardization process only produces

specification documents (Recommendations, etc), and does not formally address compliance testing.

Apache Each project defines its own testing practices and

standards

WS-I Testing Working Group2

Deliverables  The Test Tools Development Working Group

will produce the following deliverables: 1.  WS-I Test Methodology White paper 2.  WS-I Test Tools Specification 3. Two or more implementations of Test Tools and

supporting documentation 4.  Assertions and Test Conditions used as input to

the Test Tools 5.   WS-I Tool Configuration Template 6. WS-I Experience Report: Tool Development as a

component of WS-I Profile Development

2WS-I Testing Working Group Charter, http://www.ws-i.org/docs/charters/Test_Charter.pdf

Scope of the Task1

Derive testable assertions from the specificationDetermine test cases implied by each assertionDesign, develop, and test the tests for each assertionCreate a test framework to run, evaluate, and report on the tests

1TCK Project Planning and Development Guide, Sun Microsystems, Inc

Existing Frameworks

JUnit -- open source projectBaRT – Batch generator for Regression Tests -- IBM open source tool used for testing JVMs. Sun’s Compatibility Test Toolkit – licensed, limited to JCP projects

Testing Optional Functionality

Separate, or at least configurable, test suites for discrete levels of functionThe discrete levels of function currently mentioned are Simple and SophisticatedFiner distinctions may have to be madePerhaps the test suite can determine at runtime which level of function the Producer supports and be able to apply the correct tests.