ConformanceConformance

Lynne S. Rosenthal

Mary Brady

National Institute of Standards and TechnologyLynne.rosenthal@nist.gov

Mary.brady@nist.gov

OutlineOutline

Terminology Testing Methodologies Benefits of Conformance Testing Test Suites for XML Technologies

– XML, DOM, XSLT,– XSL FO, XML Schema Conformance Testing

Certification

Certificates Branding

Conformanceclause

Test suites Test tools

Validation

Procedures, reporting

Unscrambling the TerminologyUnscrambling the Terminology

Conformance

Specification Conformance Clause

Conformance Testing test suite, test tools

Validation process, reporting

Certification certificate, brand

Terminology - ConformanceTerminology - Conformance

CONFORMANCE - the fulfillment of a product, process or service of specified requirements (ISO Guide 2)

– These requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification

CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance

Terminology - Conformance TestingTerminology - Conformance Testing

CONFORMANCE TESTING – a way to verify

implementations of a specification to determine

whether or not deviations from the specifications exist (through the use of test suites)– tests capture the technical requirements within the specification– testing is bound in scope by the specification – internal structure of product-under-test not accessible (black box

testing)

Necessary, but not sufficient, for interoperability

Note: referred to as conformity assessment in ISO Guide 2

Testing MethodologyTesting Methodology

Falsification Testing– find errors by means of experimentation– show presence of errors not their absence– prove non-conformance; can never prove conformance

Tests Based on:– required functionality

– logical errors by misunderstanding requirements

– errors from boundary conditions and divergence

– common programming errors

OutputInput

Conformance Tests

Requirements

Test programs

Specification

Implementation

Terminology - Testing ProcessTerminology - Testing Process

VALIDATION - process necessary to perform conformance testing in accordance with a prescribed procedure and official test suite– ensures that testing can be repeatable and reproducible– ensures that conclusions are consistent with facts presented

in the evaluation

CERTIFICATION - acknowledgement that a validation was completed and the criteria established by CIO for issuing certificates (brands) was met.

Testing IngredientsTesting Ingredients

Standard Conformance clause

Conformance TestingTest assertions

Test suite (test software, test scripts, test criteria)

Validation Process - policy and procedures for testing

Certificationqualified bodies to do the testing and certification

control board - advisory and arbiter

Benefits of Conformance TestingBenefits of Conformance Testing

Increased probability that products are implemented correctly– Contains required functionality– Behaves as expected– Performs functions in a known manner

Increased likelihood of portability and interoperability – Portability - move software or applications among different

systems – Interoperability – ability of 2+ systems to exchange and use

information

Benefits of Conformance Testing Benefits of Conformance Testing cont.cont.

For Developers of Specifications– Feedback loop

For Software Developers– Help to improve quality - identify areas in which

they conform or not conform For Buyers

– Increases buyer’s confidence in a product and its ability to meet their needs

– Substantiate seller’s claims– Not locked into purchasing from a single vendor

Components of a Test SuiteComponents of a Test Suite

Test Harness

Test Cases

Test Scenario

Test Assertions, Axioms, Semantic Requirements

Specification

OASIS XML Conformance OASIS XML Conformance

XML Conformance Committee– Develop tests to improve the quality of XML

processors– Provide a forum to discuss issues related to

interoperability– Provide input to W3C regarding errors/ambiguities

Deliverables– Test Suite containing over 2000 tests, Second Edition– Associated test report– Built using XML/XSLT

XML Test SuiteXML Test Suite

Types of XML Tests

– Binary

• Validating processors must check validity constraints.

• Non-validating processors do not have to read external entities. Accordingly, they may not report errors which would have been detected had those entities been read.

– Output Tests

• Requirements on XML processors to ensure that they pass the necessary information on to applications.

XML Test MatrixXML Test Matrix

Non-validating

ValidatingExternal Entities Ignored

External Entities Read

Valid Documents Accept Accept Accept

Invalid Documents Accept Accept Accept

Not-wf Documents Reject Reject Reject

WF Errors tied to External Entity

Accept (varies) Reject Reject

Documents with Optional Errors

(not specified) (not specified) (not specified)

XML Test SuiteXML Test Suite

Conformance.xml

XML Test Description FilesTest Case DTD

Testcases.dtd• Test Profile

• Test URI

• Unique ID

• Spec Section

• Type

• Entities

• Description

• NIST / OASIS

• SUN Microsystems

• James Clark

• FUJI Xerox

• IBM

Test Report•XSL Stylesheets•IE5, HTML renditions

<TESTCASES PROFILE=“SUN Microsystems XML Tests”><TEST URI=“sun/invalid/e101.xml” ID=“e101” SECTIONS=“3 [39]” TYPE=“invalid” ENTITIES=“none”>Tests the Element Valid VC by including an undeclared child element.</TEST></TESTCASES>

XML.COM• Java application• Tests XML processors• Dave Brownell

Outputs

XML Test Suite Results, v1XML Test Suite Results, v1

XML.COM Conformance Update – Dave Brownell Validating Parsers

– Tested JAXP 1.0, IBM Xerces/Java 1.0.3, Layered Validator (distributed with Aelfred 2), MSXML3 Technology Preview, Oracle V2.0.2.7

– Pass Rate 82% - 99% Non-validating Parsers

– Tested all of the above plus XP 0.5– Pass rate 86% - 99%

Microsoft Update – July 2000 MSXML 3.0 Beta – Chris Lovett– Non-validating: DOM: 95%, SAX: 97%– Validating: 97%

NIST DOM Test SuiteNIST DOM Test Suite

Test Harness – automatically runs available tests and presents user witha color-coded table of results; gives access to spec, assertions, and source code

Test ScenarioLoad appropriate XML or HTML file, exercise DOM calls

Test Assertions – Fundamental, Extended, HTML Interfaces

DOM L1 Specification

800 ECMAScript fundamental, extended, HTML

200 Java – fundamental, extended

Test Cases

DOM Test SuiteDOM Test Suite

Test Results:– Run tests against IE5, Mozilla (some fundamental

interfaces), Xerces, JAXP, and Oracle; Pass rate: 82 – 98%– Implementations differ greatly in the way that parsers handle

white-space and entity expansion issues – differences in DOM tree

DOM Level 2– Working with W3C to identify resources – Modified Java tests to be consistent with SUN contribution– Will update NIST tests to L2 status

XSLT Test SuiteXSLT Test Suite

OASIS Technical Committee, chaired by G. Ken Holman

Member contributions include Lotus, NIST, Sun

Expands on XML Test Description file Dave Marston, XSLT/XPATH Conformance

Wed, 4pm, Delaware Room Public Information Meeting, Wed, 8pm, Suite

8228

XSL FO Test SuiteXSL FO Test Suite

W3C WG to test Candidate Recommendation Develop test assertions for all basic formatting

objects and properties Develop test DTD and associated harness Work with vendors to categorize available tests Research translating available CSS and XSLT

tests Develop tests for additional spec components Expected March 2001

XML Schema Test SuiteXML Schema Test Suite

Test Control

• Test Types• # schemas• # instances

Test Control

• Test Types• # schemas• # instances

Constraints

• Structure• Limits• Special info

Constraints

• Structure• Limits• Special info

Schema Tests

Test GeneratorTest Generator

SummarySummary

Interesting URL’s– http://www.nist.gov/xml/– http://www.w3.org/– http://www.oasis-open.org/

Questions???