Software Testing Conformance Diagnostics And Mark W. Skall, Division Chief Candy Leatherman,...

Software

TestingConformanceDiagnostics And

Mark W. Skall, Division ChiefMark W. Skall, Division Chief

Candy Leatherman, SecretaryCandy Leatherman, Secretaryemail: [email protected]: [email protected]

Tel. No.: 301-975-3262Tel. No.: 301-975-3262

Div. Web Site: http://www.itl.nist.gov/div897/Div. Web Site: http://www.itl.nist.gov/div897/

Program ReviewProgram Review

Division 897 Funding - FY00

STRS - $5,207K

ITS - $585K

OA - $1,615K

Employees: 35 Full Time 3 Part Time 10 Intermittent 5 Guest Researchers

Projected Balance:Projected Balance:$257,062$257,062

(+ $637,570 transferred to (+ $637,570 transferred to 2001 OA Reserve)2001 OA Reserve)

SDCT’sSDCT’s GOALGOAL

To improve quality of software in industry through the development of:To improve quality of software in industry through the development of: TechnologyTechnology MeasurementsMeasurements StandardsStandards

Specific Strategy:Specific Strategy: Conformance TestsConformance Tests Reference ImplementationsReference Implementations Diagnostic TestsDiagnostic Tests Research to reduce costs of testingResearch to reduce costs of testing Reference DataReference Data Standards developed jointly with Standards developed jointly with industryindustry

Philosophy:Philosophy: Concentrate on key areas at forefront of technologyConcentrate on key areas at forefront of technology Get involved early and partner with industryGet involved early and partner with industry Fill industry voidFill industry void Transfer technology and move onTransfer technology and move on

Excerpts from Assessment Panel Report – Excerpts from Assessment Panel Report – 6/26/006/26/00

Division ReviewDivision Review

““The planning and documentation methods used by SDCT could serve as a The planning and documentation methods used by SDCT could serve as a model for other ITL divisions. The process has used clearly-defined criteria to model for other ITL divisions. The process has used clearly-defined criteria to select state-of-the-art programs with clearly defined priorities and goals, well-select state-of-the-art programs with clearly defined priorities and goals, well-identified roles, and measurable contributions to national and international identified roles, and measurable contributions to national and international standards organizations. Examples include the work on conformance tests for standards organizations. Examples include the work on conformance tests for XML, Distributed Interactive Learning Systems, and for DASE . . .Directions XML, Distributed Interactive Learning Systems, and for DASE . . .Directions can change quickly within the software industry. The division carefully can change quickly within the software industry. The division carefully monitors these changes and acts appropriately. . . .Industry has provided many monitors these changes and acts appropriately. . . .Industry has provided many testimonials concerning the great value of the division’s standards work, testimonials concerning the great value of the division’s standards work, including: ‘The OASIS-NIST XML Conformance test suite is critical for our including: ‘The OASIS-NIST XML Conformance test suite is critical for our industry’, Norbert Mikula, CTO, OASIS; ‘NIST made strong contributions industry’, Norbert Mikula, CTO, OASIS; ‘NIST made strong contributions (X3D) and resolved our knottiest problems’, Don Brutzman, Board of (X3D) and resolved our knottiest problems’, Don Brutzman, Board of Directors, Web3D consortium, and ‘The high quality test suite and certification Directors, Web3D consortium, and ‘The high quality test suite and certification program is an invaluable resource for ATA’, Robert Peel, Director of program is an invaluable resource for ATA’, Robert Peel, Director of Airworthiness and Standards”Airworthiness and Standards”

Excerpts from Assessment Panel Report – Excerpts from Assessment Panel Report – 6/26/006/26/00

Laboratory Level ReviewLaboratory Level Review

““New projects are startedNew projects are started based on the importance of the work to U.S. industry, based on the importance of the work to U.S. industry, and the work viewed by the panel generally had specific, focused goals. An and the work viewed by the panel generally had specific, focused goals. An example of this approach is the work on XML done in SDCT. This project is example of this approach is the work on XML done in SDCT. This project is notable because it addresses a major standard on which industry was making notable because it addresses a major standard on which industry was making little progress, and it also effectively leveraged skills and approaches developed little progress, and it also effectively leveraged skills and approaches developed in a now terminated project on VRML. . .Another example of the improved in a now terminated project on VRML. . .Another example of the improved planning process is the existence of clear termination criteria, which have been planning process is the existence of clear termination criteria, which have been useful in sunsetting projects. For example, the Real-Time Java project and the useful in sunsetting projects. For example, the Real-Time Java project and the work on RBAC satisfied their completion criteria and therefore ended this work on RBAC satisfied their completion criteria and therefore ended this year.”year.”

Standards and Conformance Testing GroupStandards and Conformance Testing Group

Software Quality GroupSoftware Quality Group

Interoperability GroupInteroperability Group

Software Diagnostics and Conformance Testing

• Development of conformance testsDevelopment of conformance tests

• Development of reference implementationsDevelopment of reference implementations

• Research into better ways to do conformance testingResearch into better ways to do conformance testing

• Develop standards jointly with industryDevelop standards jointly with industry

Standards andConformance Testing Group

Group Leader: Lynne RosenthalGroup Leader: Lynne RosenthalSupy Proj. Leader: Lisa CarnahanSupy Proj. Leader: Lisa Carnahan

Clare Lucey, SecretaryClare Lucey, Secretary

Standards andConformance Testing Group

Mary BradyMary BradyLisa CarnahanLisa CarnahanLaurent Ciarletta (GR)Laurent Ciarletta (GR)Anthony Cincotta (S)Anthony Cincotta (S)Chris DabrowskiChris DabrowskiAlden DimaAlden DimaLeonard GallagherLeonard GallagherLeonard GebaseLeonard GebaseNeil Gima (S)Neil Gima (S)Alan GoldfineAlan Goldfine

Martha GrayMartha GrayMichael KassMichael KassClare LuceyClare LuceyThomas LogueThomas LogueCarmelo Montanez-RiveraCarmelo Montanez-RiveraThomas RhodesThomas RhodesRichard RivelloRichard RivelloJacqeline SchneiderJacqeline SchneiderJohn TebbuttJohn TebbuttMarie-Noelle Terasse (GR) Marie-Noelle Terasse (GR) Mark ZimmermanMark Zimmerman

Division 897Standards and Conformance Testing

Group

• Develop methods to automate software testingDevelop methods to automate software testing

• Develop software diagnostic toolsDevelop software diagnostic tools

• Develop reference dataDevelop reference data

• Formal methodsFormal methods

Software Quality Group

Group Leader: John BarkleyGroup Leader: John BarkleyDebbie Blackstone, SecretaryDebbie Blackstone, Secretary

Software Quality Group

Tamer Ahmed Tamer Ahmed (GR)(GR)Paul Ammann (F)Paul Ammann (F)Debbie BlackstoneDebbie BlackstoneDavid Brinkley (F)David Brinkley (F)Paul BlackPaul BlackNeva CarlsonNeva CarlsonJohn CherniavskyJohn CherniavskyAnthony CincottaAnthony Cincotta

Keith Gallagher Keith Gallagher (F)(F)Roger Gima (S)Roger Gima (S)Michael KooMichael KooMary LaamanenMary LaamanenJames LyleJames LyleWilliam MajurskiWilliam MajurskiDouglas WhiteDouglas White

Division 897Software Quality

Group

• Ensure Federal agency requirements are input into voluntary standards Ensure Federal agency requirements are input into voluntary standards committees - work through Federal CIO Councilcommittees - work through Federal CIO Council

• Help Federal agencies and industry achieve interoperability through Help Federal agencies and industry achieve interoperability through application of Division productsapplication of Division products

• Provide technical support to voluntary standards committeesProvide technical support to voluntary standards committees

• Support NIST paperless office effort utilizing digital signaturesSupport NIST paperless office effort utilizing digital signatures

InteroperabilityGroup

Acting Group Leader: Lisa CarnahanActing Group Leader: Lisa CarnahanVacant, SecretaryVacant, Secretary

InteroperabilityGroup

Daniel AllenDaniel AllenBruce Bargmeyer (RA)Bruce Bargmeyer (RA)Frederick BolandFrederick BolandGary FisherGary FisherLarry Fitzwater (GR)Larry Fitzwater (GR)

Elizabeth FongElizabeth FongKathryn HarvillKathryn HarvillRoy MorganRoy MorganJudith NewtonJudith NewtonGertrude Gertrude SherwoodSherwood

Division 897Interoperability

Group

StrategyStrategy

•Conformance TestsConformance Tests

Reference ImplementationsReference Implementations

Standards developed jointly with industryStandards developed jointly with industry

Diagnostic TestsDiagnostic Tests

Research to reduce costs of testingResearch to reduce costs of testing

Reference DataReference Data

Development of Conformance TestsDevelopment of Conformance Tests

• Emphasis on newer technologiesEmphasis on newer technologies

• Tests developed in parallel with standardsTests developed in parallel with standards

• Tests developed before implementationsTests developed before implementations

• Need buy-in from industryNeed buy-in from industry

• Need high impactNeed high impact

• Need technology transferNeed technology transfer

Development of Conformance Tests (cont’d)Development of Conformance Tests (cont’d)

• Legacy activitiesLegacy activitiesCGMCGMSQLSQLPOSIXPOSIXAda and other programming languageAda and other programming languageVRMLVRML

• CurrentCurrent activities activitiesXML and companion standards (DOM, XSL, etc.) XML and companion standards (DOM, XSL, etc.) Digital TV Digital TV X3DX3DLearning Technologies (IMS)Learning Technologies (IMS)

XML Conformance TestingXML Conformance Testing

Project Team:Project Team:

Mary BradyMary BradyCarmelo MontanezCarmelo MontanezRicky RivelloRicky RivelloMark ZimmermanMark Zimmerman

Electronic CommerceElectronic Commerce

XML Technologies

B2C(business-to-consumer)

InteractiveTelevision

(DASE, SMPTE)

Internet / World Wide Web

B2B(business-to-business)

Manufacturing(ICM)

DistributedLearning

(IMS)

Electronic Commerce Growth– Growth is exponential, already outpacing last year’s predictions– Changing the way businesses do business– Revolutionizing the way we consume and play

Advances are necessary in:– Tools for describing / sharing information

XML technologies

– Applying IT to solve vertical market demands NIST provides:

– XML Technologies Conformance Testing

– NIST developed tests, available on-line, for use in testing applications– NIST technical leadership, neutral third-party – Unbiased Feedback of Specification Errors

– Vertical Markets Applied IT research

Why NIST?

HTML or XML?

Figure 1: HTML Purchase Order

<H1>Purchase Order</H1><H1>Purchase Order</H1>

<UL><UL>

<LI><b> Mary Brady</b> <LI><b> Mary Brady</b>

<LI>NN Rm 572, Gaithersburg, MD, 20899<LI>NN Rm 572, Gaithersburg, MD, 20899

<LI>1234<LI>1234

<LI><I>Trees</I><LI><I>Trees</I>

<LI>50.00<LI>50.00

<LI>20<LI>20

<LI>1000.00<LI>1000.00

<LI>Be sure to water the root ball day of <LI>Be sure to water the root ball day of deliverydelivery

<LI>Susan Carscadden<LI>Susan Carscadden

<LI>NIST, Bldg 301, Gaithersburg, MD, <LI>NIST, Bldg 301, Gaithersburg, MD, 2089920899

</UL</UL

What is XML?•Domain-specific languages•Data separate from display•Self-describing data•Schemas

<Purchaser><Purchaser><Name> Mary Brady </Name><Name> Mary Brady </Name><Address Street=NN Rm 572 <Address Street=NN Rm 572

City=Gaithersburg, State=MD City=Gaithersburg, State=MD Zip=20899</Address>Zip=20899</Address>

</Purchaser></Purchaser><Item><Item><Part_number>1234</Part_number><Part_number>1234</Part_number><ItemDescription>Trees</<ItemDescription>Trees</

ItemDescription>ItemDescription><Cost>50.00</Cost><Cost>50.00</Cost><Quantity>20</Quantity><Quantity>20</Quantity><TotalCost>1000.00</TotalCost><TotalCost>1000.00</TotalCost><Instructions>Be sure to water the root <Instructions>Be sure to water the root

ball day of delivery</Instructions>ball day of delivery</Instructions></Item></Item><ShipTo><ShipTo><Name>Susan Carscadden</Name><Name>Susan Carscadden</Name><Address Street=NIST, Bldg 301 <Address Street=NIST, Bldg 301

City=Gaithersburg State=MD City=Gaithersburg State=MD Zip=20899</Address>Zip=20899</Address>

</ShipTo></ShipTo>

Figure 2: XML Purchase Order

XML TechnologiesXML Technologies

XML Syntax, Information Set, Schema, Linking, Fragment

Registries Repositories

Namespaces

Programming Languages

DOM

Electronic Commerce, Distance Learning, Health CareManufacturing, Interactive Televison

TransformationsFormatting Objects

Stylesheets Query Language

Signatures

Information Description

Ratings Libraries Control

Domain Specific

Boa

rds

Med

ical

Edu

catio

n

Pro

cure

men

t

...Aut

o P

arts XML.ORG

BIZTalk

E-BusinessLearning Objects

XML/EDI

... ...


OASIS XML ConformanceOASIS XML Conformance Sun, IBM, Fuji Xerox, Sun, IBM, Fuji Xerox,

DataChannel, MicroStar, DataChannel, MicroStar, W3C membersW3C members

Why NIST?Why NIST? Technical LeadershipTechnical Leadership Coalesced industry Coalesced industry

partnerspartners18 months

First DeliverableFirst Deliverable XML Test SuiteXML Test Suite XML files & XSL XML files & XSL

StylesheetStylesheet6 months

XML Test Suite•1000 XML Tests

•DTD + 4000 lines of XML•400 lines of XSL Stylesheet

Future WorkFuture Work XML StyleSheets, XML StyleSheets,

SchemasSchemas XML NS, XLink, XPtrXML NS, XLink, XPtrContinued Support

http://xw2k.sdct.itl.nist.gov/xml/conf/xmlconf.xml

http://bogey5.ncsl.nist.gov/xml/tests/conformance/conformance.xml

DOM Conformance TestingDOM Conformance TestingW3C, OASISW3C, OASIS

Sun, IBM, Microsoft, Sun, IBM, Microsoft, Netscape, OracleNetscape, Oracle

XML ‘98XML ‘98 Met with W3C WG ChairMet with W3C WG Chair NIST asked to develop NIST asked to develop

teststestsXML ‘98

NIST ReleasedNIST Released Fundamental, Extended Fundamental, Extended

(ECMAScript)(ECMAScript) Interactive test harnessInteractive test harnessJune ‘99

DOM Test Suite• 300 ECMAScript Tests

• 14,500 lines of code

Future WorkFuture Work Fundamental, Extended Fundamental, Extended

(Java)(Java) HTML (ECMAScript & Java)HTML (ECMAScript & Java) DOM Level 2DOM Level 2

Continued Support

http://xw2k.sdct.itl.nist.gov/dom/index.html

http://bogey5.ncsl.nist.gov/xml/tests/conformance/conformance.xml

FY00 Progress -TestsFY00 Progress -Tests Completed XML Test Suite (release 1)

Completed XML Test Suite (release 2)

– Incorporated new XML tests and fixes Completed DOM-Ecmascript Test Suite

(release 1) Completed DOM-Java Test Suite (release

1) Develop XSLT Test Suite

– partnership with Lotus Corp. and OASIS

FY00 Progress -Committee FY00 Progress -Committee LeadershipLeadership

Chair OASIS Conformance and XML Chair OASIS Conformance and XML Testing CommitteesTesting Committees

Provide leadership for XSLT and Schema Provide leadership for XSLT and Schema Testing committeesTesting committees

Provide leadership for UN/CEFACT Provide leadership for UN/CEFACT ebXML initiative (conformance and ebXML initiative (conformance and reg/rep)reg/rep)– Provide conformance guidance– Develop conformance clause – Harmonize OASIS and ebXML efforts

FY01 PlansFY01 Plans

Complete XSLT test suite Develop test suites for XML Schema

and other XML Technologies Develop XML-based automated test

tools to improve test development Continue to chair OASIS testing

committees

FY00 ResourcesFY00 Resources 3.5 FTE3.5 FTE $700K (STRS)$700K (STRS)

FY01 ResourcesFY01 Resources 5 FTE Travel required to collaborate in

standards and test development efforts Equipment: 10K


Customers:Customers:– IT industry: e.g., IBM, Sun, Microsoft, Oracle, IT industry: e.g., IBM, Sun, Microsoft, Oracle,

DataChannel, DocumentumDataChannel, Documentum

– Vertical industries: e.g., Education (IMS), Financial Vertical industries: e.g., Education (IMS), Financial (e.g., Dun and Bradstreet), Travel (e.g., Sabre)(e.g., Dun and Bradstreet), Travel (e.g., Sabre)

Impact:Impact:– Conformance test suites have been used to Conformance test suites have been used to

improve the quality of XML/DOM processors improve the quality of XML/DOM processors (many of which are embedded in other XML (many of which are embedded in other XML software solutions). These are used in many software solutions). These are used in many vertical markets by millions of customers.vertical markets by millions of customers.

– Feedback to standards developers improve the Feedback to standards developers improve the specificationsspecifications

Interactive TV Interactive TV


Alan GoldfineAlan GoldfineJohn BarkleyJohn BarkleyDoug WhiteDoug WhiteLen GebaseLen Gebase

Interactive TVInteractive TV

• TV and the Internet are each recognized as “a technology …TV and the Internet are each recognized as “a technology …that seems to change everything” (BW, Oct 4 ‘99)that seems to change everything” (BW, Oct 4 ‘99)

• Interactive TV is the convergence of TV and the InternetInteractive TV is the convergence of TV and the Internet

• Profound impact on Electronic Commerce expectedProfound impact on Electronic Commerce expected

• WebTV: 1 million subscribers, 350 hrs/wk of programmingWebTV: 1 million subscribers, 350 hrs/wk of programming

• AOLTV: projected 2 million subscribers in 2 yearsAOLTV: projected 2 million subscribers in 2 years

• ITL partnering to develop the standards and tests required ITL partnering to develop the standards and tests required for the for the success of Interactive TV success of Interactive TV

Interactive TV

• Wireless phones do emailWireless phones do email

• Internet does broadcast radio and TVInternet does broadcast radio and TV

• Broadcast TV references links to Web pagesBroadcast TV references links to Web pages

Traditionally separate media are converging:Traditionally separate media are converging:

Currently, for the viewer:Currently, for the viewer:

accessed separatelyaccessed separately}}TV:TV:

(non-interactive)(non-interactive)

OROR

Web:Web:(interactive)(interactive)


With DASE/DDE, TV and Web are converging into a With DASE/DDE, TV and Web are converging into a singlesingle medium: medium:

DASE/DDEDASE/DDE

DemonstrationDemonstration

Current production:Current production:

Web site with associated productsWeb site with associated products Golf matches on TVGolf matches on TV


DASE/DDE: DASE/DDE: singlesingle medium medium

DASE/DDEDASE/DDE

C:\Program Files\dase\dase-golf.exe

DemonstrationDemonstrationInteractive TVInteractive TV

C:\Program Files\dase\dase-golf.exe

• Members include: Members include: ABC, CBS, IBM, Intel, Lucent, ABC, CBS, IBM, Intel, Lucent, Microsoft, NBC, Warner BrothersMicrosoft, NBC, Warner Brothers

• DASE (DTV Application Programming Environment): DASE (DTV Application Programming Environment): Standard for a platform independent, high-level Standard for a platform independent, high-level abstraction for integrating DTV and the Internetabstraction for integrating DTV and the Internet

• DASE specifies programming environment for DTV DASE specifies programming environment for DTV receiversreceivers

• Downloadable DASE applications portable across Downloadable DASE applications portable across different TV set topsdifferent TV set tops

Advanced Television Systems Committee (ATSC)Advanced Television Systems Committee (ATSC)



• Joint project with the High Performance Systems and Joint project with the High Performance Systems and Services Division (895)Services Division (895)

•Chair DASE Conformance Working Group where key Chair DASE Conformance Working Group where key players include: players include: ABC, Gateway, Microsoft, SunABC, Gateway, Microsoft, Sun

• Partner with Unisoft to develop conformance testsPartner with Unisoft to develop conformance tests

• Reference implementation for programming environment Reference implementation for programming environment

NIST’s Role in ATSC DASE:NIST’s Role in ATSC DASE:

• Members include: Members include: ABC, CBS, IBM, Intel, Microsoft, ABC, CBS, IBM, Intel, Microsoft, NBC, Warner BrothersNBC, Warner Brothers

• DDE (Declarative Data Essence): Standard for Internet DDE (Declarative Data Essence): Standard for Internet content and bindings to analog and digital streams content and bindings to analog and digital streams (formerly ATVEF)(formerly ATVEF)

• Declarative Content: HTML, CSS, ECMAScript, DOM, Declarative Content: HTML, CSS, ECMAScript, DOM, Triggers (for syncing the two mediums)Triggers (for syncing the two mediums)

Society of Motion Picture and Society of Motion Picture and Television Engineers (SMPTE):Television Engineers (SMPTE):



• Contribute text to specificationContribute text to specification

• Partner with Unisoft to develop conformance tests Partner with Unisoft to develop conformance tests

• Develop conformance tests for triggers and stream binding to Develop conformance tests for triggers and stream binding to integrate with existing Unisoft DDE element testsintegrate with existing Unisoft DDE element tests

• Develop prototype receiver testbedDevelop prototype receiver testbed

• Participate in Applications Data Essence (ADE) Study GroupParticipate in Applications Data Essence (ADE) Study Group

NIST’s Role in SMPTE DDE:NIST’s Role in SMPTE DDE:

FY00 ProgressFY00 Progress Chair DASE conformance task groupChair DASE conformance task group Conformance section of the DASE Conformance section of the DASE

Standard completedStandard completed Review of DASE conformance test Review of DASE conformance test

assertions from Unisoft completedassertions from Unisoft completed DDE Standard completed, adoption DDE Standard completed, adoption

underwayunderway DDE Bindings Standard underwayDDE Bindings Standard underway Test assertions for DDE triggers underwayTest assertions for DDE triggers underway Testbed for DDE receivers underwayTestbed for DDE receivers underway ADE Study draft completedADE Study draft completed


Continue to chair DASE conformance Continue to chair DASE conformance groupgroup

DASE Standard adoptedDASE Standard adopted Continue DASE conformance test Continue DASE conformance test

development with Unisoftdevelopment with Unisoft DDE and DDE bindings Standards adoptedDDE and DDE bindings Standards adopted Develop DDE conformance tests for Develop DDE conformance tests for

triggers, integrate with Unisoft tests triggers, integrate with Unisoft tests Develop test assertions for DDE bindingsDevelop test assertions for DDE bindings Complete DDE Receiver testbedComplete DDE Receiver testbed Begin DDE-2 Standard developmentBegin DDE-2 Standard development

FY00 ResourcesFY00 Resources 2 FTE2 FTE $400K (STRS, ATP-$100K)$400K (STRS, ATP-$100K)

FY01 ResourcesFY01 Resources 4.5 FTE4.5 FTE Equipment: $20KEquipment: $20K Travel required for standards Travel required for standards

meetings and testing meetings and testing collaborationscollaborations

Interactive TVInteractive TV CustomersCustomers

– ITIT, e.g., IBM, Microsoft, e.g., IBM, Microsoft– broadcast TVbroadcast TV, e.g., NBC, DirecTV, e.g., NBC, DirecTV– entertainmententertainment, e.g., Disney, Universal, e.g., Disney, Universal– consumer electronicsconsumer electronics, e.g., RCA, Intel, e.g., RCA, Intel

ImpactImpact– New Medium: TV/Internet convergenceNew Medium: TV/Internet convergence– Interactive applications portable across receivers from different Interactive applications portable across receivers from different

manufacturersmanufacturers– ATSC and SMPTE invited NIST participationATSC and SMPTE invited NIST participation– NIST chairs DASE Conformance GroupNIST chairs DASE Conformance Group

X3D Conformance TestingX3D Conformance Testing

Project Leader:Project Leader:

Mike KassMike Kass

ObjectivesObjectives Continue the NIST/Web3D partnership begun in 1996Continue the NIST/Web3D partnership begun in 1996

– ITL created the VRML Test Suite (VTS) and Viper parserITL created the VRML Test Suite (VTS) and Viper parser Create an X3D Test Suite ( XTS )Create an X3D Test Suite ( XTS )

– Modify VRML test requirements and test descriptionsModify VRML test requirements and test descriptions– Translate existing VRML tests to X3D formatTranslate existing VRML tests to X3D format

Chair the Interoperability/Conformance WGChair the Interoperability/Conformance WG– Interface with developers and specification writers to resolve ambiguities in specificationInterface with developers and specification writers to resolve ambiguities in specification– Provide feedback to developers on browser conformanceProvide feedback to developers on browser conformance

FY00 ProgressFY00 Progress

Completed translation of VTS to X3D Test Suite (XTS)

Developed test harness to enable developers to contribute tests

Interest from DraW and Sun for using the test suite and making the results public


Low level support - assisting community in using the VRML and X3D Test Suites

No new development planned. Maintenance as needed

FY00 ResourcesFY00 Resources 0.75 FTE $150K (STRS)

FY01 Resources 0.2 FTE (minimal) No Travel is expected

X3D Conformance TestingX3D Conformance Testing

Customers:Customers:– X3D Consortium including Sun, Sony X3D Consortium including Sun, Sony

DraW Computing, Shout, BlaxxunDraW Computing, Shout, Blaxxun

Impact:Impact:– Conformance test suites are being Conformance test suites are being

used to improve the quality of X3D used to improve the quality of X3D browsersbrowsers

– DraW Computing has said that they DraW Computing has said that they would make their test results publicwould make their test results public

Learning TechnologiesLearning Technologies

Project team:Project team:Martha GrayMartha GrayTom RhodesTom RhodesJacki SchneiderJacki SchneiderLisa CarnahanLisa Carnahan

Learning TechnologiesLearning TechnologiesLearning technology is: Learning technology is: Learning anytime, anyplaceLearning anytime, anyplace Interactive, distributive learning systems Interactive, distributive learning systems

In collaboration with industry, NIST/ITL is:In collaboration with industry, NIST/ITL is: defining requirements and specificationsdefining requirements and specifications developing prototypes and reference implementationsdeveloping prototypes and reference implementations developing testing techniques and toolsdeveloping testing techniques and tools

Why Now?Why Now? ““The Gartner Group estimates that companies in the United The Gartner Group estimates that companies in the United

States spent $58 billion on training in 1998, and of that, $9.4 States spent $58 billion on training in 1998, and of that, $9.4 billion went to IT-related training. They expect this figure to billion went to IT-related training. They expect this figure to grow to $10.5 billion this year.” grow to $10.5 billion this year.” Gilbane Report, July/Aug 1999Gilbane Report, July/Aug 1999

““Just as electronic commerce exploded over the past two Just as electronic commerce exploded over the past two years, the stage is now set for ‘e-learning’ to become the years, the stage is now set for ‘e-learning’ to become the next big wave in Internet-based applications” next big wave in Internet-based applications” Cisco Systems CEO Cisco Systems CEO and President John Chambersand President John Chambers

““Worldwide, the education and training market is over a Worldwide, the education and training market is over a trillion (US) dollars.” trillion (US) dollars.” IEEEIEEE

ProjectsProjects1.1. IMS Global Learning Consortium, Inc.IMS Global Learning Consortium, Inc.

Chair conformance teamChair conformance team– advise members on writing IMS specificationsadvise members on writing IMS specifications– ensure testability of specifications ensure testability of specifications – advise on how to test and set up testing programsadvise on how to test and set up testing programs

Develop Meta-data repository reference implementation Develop Meta-data repository reference implementation

2.2. IEEE and ISOIEEE and ISO Provide conformance expertise Provide conformance expertise Harmonization of efforts among specificationsHarmonization of efforts among specifications

3.3. DoD’s Advanced Distributive Learning (ADL)DoD’s Advanced Distributive Learning (ADL) Executive Order 13111 on Federal Training - NIST/ITL co-Executive Order 13111 on Federal Training - NIST/ITL co-

leaders with ADL and DOL. leaders with ADL and DOL.

FY00 ProgressFY00 Progress Continue collaboration with IMS, IEEE LT

committee and NIST/ATP Chair IMS conformance testing activities Developed IMS conformance test policies and

procedures Served as Technical Lead of SBIR solicitation,

assessed and recommend proposals Develop repository of learning objects (this

work coordinates with Reg/rep subproject)


Continue to chair IMS conformance committee providing expertise to emerging IMS specifications

Develop a conformance test plan for applicable IMS specifications

FY00 ResourcesFY00 Resources 1.5 FTE1.5 FTE $300K (STRS, ATP)$300K (STRS, ATP)

FY01 ResourcesFY01 Resources 1 FTE Minimal travel required to attend

standards meetings and interoperability demos

Learning TechnologiesLearning Technologies

Customers:Customers:– IMS, DoL, DoD, Simon&Schuster, Sun, IBM, Apple, IMS, DoL, DoD, Simon&Schuster, Sun, IBM, Apple,

Microsoft, UNC, Calif State UMicrosoft, UNC, Calif State U

Impact:Impact:– Use of standard metadata for global learning will Use of standard metadata for global learning will

improve discovery, evaluation, and use/reuse of improve discovery, evaluation, and use/reuse of learning objectslearning objects

– NIST’s conformance expertise contributed to NIST’s conformance expertise contributed to clearer, less ambiguous, testable IMS clearer, less ambiguous, testable IMS specificationsspecifications

– Conformance test plans and procedures will Conformance test plans and procedures will enable IMS to determine compliant productsenable IMS to determine compliant products

ATA CGM Testing ProgramATA CGM Testing Program

FY00 ProgressFY00 Progress Completed final release of the ATA CGM

Test Suite Provided guidance to ATA Recognized Test

Laboratory on using the test suite and assessing clients

Chair Control Board - resolve questions of interpretation of the test suite and standard.

Initiated discussions on ATA's eCommerce activities - explore a possible XML testing program


Continue to support ATA's CGM Test Program– No new CGM development planned– Maintenance as needed

Provide guidance on the use of NIST's XML test suite and its applicability and use in ATA


FY01 ResourcesFY01 Resources 0.25 FTE Travel required to support ATA test

program and potential for an ATA XML test program

CGMCGM

Customers:Customers:– Air Transport Association – Airlines and their Air Transport Association – Airlines and their

suppliers (e.g., United, Northwest, Airbus, suppliers (e.g., United, Northwest, Airbus, FedX, Boeing, Rolls-Royce)FedX, Boeing, Rolls-Royce)

– CGM Developers (e.g., Autotrol, Intergraphics) CGM Developers (e.g., Autotrol, Intergraphics)

Impact:Impact:– NIST Conformance test suite used by ATA NIST Conformance test suite used by ATA

CGM Test Program to validate CGM CGM Test Program to validate CGM interpretersinterpreters

– Quality, reliable CGM products used for Quality, reliable CGM products used for representing graphics in airframe representing graphics in airframe maintenance manualsmaintenance manuals

Operational Testing IssuesOperational Testing Issues

Test suites should be available on the InternetTest suites should be available on the Internet

Test suites should be free of chargeTest suites should be free of charge

Operational testing should be done by private sectorOperational testing should be done by private sector

Industry associations may issue certificatesIndustry associations may issue certificates

Technology Transfer from NIST to ATA

Developed CGM Interpreter Test Suite

Developed Test Program FrameworkDeveloped test procedures Formed and chair Control BoardDeveloped Test Lab criteria and evaluation criteria

Assessed Test Lab for ATA recognition

Serve as technical adviser

Completed transfer

Testing ServicesStatus of Software Operational Testing Services

TESTSERVICE

STARTDATE

TERMINATEDATE

CERTIFICATEOFFERED BY:

TESTING OFFEREDBY:

NSTL NSTLTerwilliger Terwilliger

SQL 1990 July 1 1997

EDS EDSIEEE Mindcraft, Perennial

(NVLAPs)POSIX 1991 Dec 31 1997

X/Open X/OpenAda 1985 July 1997 AJPO Recognized Test

LabsFortran78 1979 (GSA)

1986 (NIST)June 7 1998 EDS EDS

Cobol 85 1974 (Navy)1986 (NIST)

June 7 1998 EDS EDS

C 1989 Oct 1 1998 EDS and/orPerennial

EDS and/or Perennial

CGM 1994 Oct 1 1998 ATA ATA recognized TestLabs

Testing Services (cont’d)Testing Products or Services - Offered by Other Organizations - Developed with NIST Assistance

TEST SERVICE

START DATE

SPONSORED BY TYPE OF SERVICE

VRML 1997 VRML Consortium (VRMLC)

Self testing of browsers using NIST VTS. Self testing of VRML Content using NIST Viper

VRMLC is discussing a certification program

Spatial Data Transfer (TVP)

1998 US Geological Survey (USGS)

Test implementations (encoders, decoders, transfers) of SDTS

plans for EDS to issue Certificates and do testing

IMS 1999 IMS Test implementations of IMS specifications and prototypes

working with IMS to develop framework for certification program

StrategyStrategy

Conformance TestsConformance Tests

•Reference ImplementationsReference Implementations Standards developed jointly with industryStandards developed jointly with industry Diagnostic TestsDiagnostic Tests Research to reduce costs of testingResearch to reduce costs of testing Reference DataReference Data

Reference ImplementationsReference Implementations

3 Purposes3 Purposes

Used as a metric of comparison in conformance testingUsed as a metric of comparison in conformance testing

Used to sort out errors and ambiguities in standardsUsed to sort out errors and ambiguities in standards

Used as a prototypeUsed as a prototype

AromaPervasive Computing Project

Project Team:Project Team:Alden DimaAlden DimaBill MajurskiBill MajurskiMary LaamanenMary Laamanen

AromaAromaPervasive Computing ProjectPervasive Computing Project

Emerging trend towardEmerging trend toward– numerous, easily accessible, often inconspicuous numerous, easily accessible, often inconspicuous

computing devices and sensors; mobile or embedded in computing devices and sensors; mobile or embedded in the environment; connected to ubiquitous network the environment; connected to ubiquitous network structurestructure

Industry is asking forIndustry is asking for– Standards for service discovery, APIs, e-commerce servicesStandards for service discovery, APIs, e-commerce services– Testing and diagnostic tools to improve quality, Testing and diagnostic tools to improve quality,

interoperabilityinteroperability– Reference implementationsReference implementations

Aroma = multi-division effort to:Aroma = multi-division effort to:– Explore standardization and measurement issuesExplore standardization and measurement issues– Attract industry interest and partnershipsAttract industry interest and partnerships

Aroma PrototypeAroma Prototype (Aroma adaptor + Aroma projector)

Goals - To discover technical, standards and measurement issues Goals - To discover technical, standards and measurement issues related to:related to:– Connecting portable wireless devices to traditional networksConnecting portable wireless devices to traditional networks– Service discovery, self-configuration and dynamic resource sharingService discovery, self-configuration and dynamic resource sharing– Mobile code and dataMobile code and data– Software infrastructure needed to create and manage pervasive Software infrastructure needed to create and manage pervasive

services and applications services and applications


Present papers at NIST PC 2000, JavaOne, and International Pervasive Computing Workshop

CCo-chaired NIST Pervasive Computing 2000 conference

Developed Aroma Adapter-Projector with Div. 892

Measured performance of Aroma adapter Developed conceptual model based on the OSI

Reference Model Exploring low-end pervasive computing

– Embedded and 8,16-bit processors– Collaboration with US Naval Academy


Co-chair NIST Pervasive Computing 2001 conference

Explore use of ADL in pervasive computing model

Implement a virtual private smart space (VPSS) test bed

Extend Aroma projector to include digital cameras and VPSS technology

Continue performance measurements Develop diagnostic and testing tools

FY00 ResourcesFY00 Resources

1.50 FTE $300K (STRS, OA-DISA)

FY01 Resources 3 FTE Travel required to conferences to present

work and collaborate with colleagues Equipment: $15K

Pervasive ComputingPervasive ComputingCustomers:Customers:

– IT industry: the usual suspectsIT industry: the usual suspects

– Consumers: e.g., GE, Sunbeam, Phillips, Northeast Consumers: e.g., GE, Sunbeam, Phillips, Northeast Utilities, XeroxUtilities, Xerox

– Universities: USNA, UMD, UC Berkeley Universities: USNA, UMD, UC Berkeley

Impact:Impact:– Solicited industry needs for standards and Solicited industry needs for standards and

measurementsmeasurements

– PC 2000 conference launched awareness of NIST as PC 2000 conference launched awareness of NIST as key playerkey player

– Papers presented at conference stimulated Papers presented at conference stimulated discussions on pc and standardsdiscussions on pc and standards

– Ref imp demonstrated feasibility of PCRef imp demonstrated feasibility of PC

– Enable widespread adoption and integration with Enable widespread adoption and integration with software tools to create, manage, measure and test software tools to create, manage, measure and test pervasive services and applicationspervasive services and applications

XML Registry/RepositoryXML Registry/Repository(Reg/Rep)(Reg/Rep)


Lisa CarnahanLisa CarnahanLen GallagherLen GallagherMike KassMike Kass

Registry

DISCOVERY

RETRIEVAL

Internet

Registry

XML Registry/RepositoryXML Registry/RepositoryReference ImplementationReference Implementation

Built to OASIS specificationBuilt to OASIS specification Being used by both OASIS and Being used by both OASIS and

UN/CEFACT (ebXML) UN/CEFACT (ebXML) Built using IMS Meta-data Built using IMS Meta-data

specificationspecification Apply lessons learned to EC Apply lessons learned to EC

repositoriesrepositories


Revised the OASIS Registry and Repository Specification

Implement a reference implementation

Participate in ebXML Reg/Rep team– Ensure testability of specification– ensure compatibility between ebXML

and OASIS specifications


Update reference implementation Develop conformance test

suite/data for OASIS specification Coalesce the ebXML and OASIS

specifications

FY00 ResourcesFY00 Resources 1.5 FTE $325K (STRS)

FY01 Resources 2 FTE Travel required to attend

standards meetings

XML Registry/RepositoryXML Registry/Repository CustomersCustomers

– EC/Enterprise Integrators: Data Channel, Documentum, EC/Enterprise Integrators: Data Channel, Documentum, Oracle, Microsoft, Sun, IBMOracle, Microsoft, Sun, IBM

– Consumers: Vertical market partnerships and supply-Consumers: Vertical market partnerships and supply-chains (e.g., automotive, insurance, travel, chains (e.g., automotive, insurance, travel, manufacturing)manufacturing)

ImpactImpact– Repositories allow efficient use of XML and eliminate Repositories allow efficient use of XML and eliminate

duplicationduplication– Enable SMEs to participate in EC marketplaces and Enable SMEs to participate in EC marketplaces and

supply chainssupply chains– Reduce cost of doing business through timeliness and Reduce cost of doing business through timeliness and

increased automationincreased automation

Internet Commerce for Internet Commerce for Manufacturing (ICM)Manufacturing (ICM)

(ITL,EEEL, MEL, MEP)(ITL,EEEL, MEL, MEP)

Project team:Project team:Tom RhodesTom RhodesLiz FongLiz FongKathy HarvillKathy Harvill

FY00 ProgressFY00 Progress Assumed project leadership (at request of EEEL) Defined standards-based framework for ICM testbed

(reference implementation) Conducted workshops and presented papers/demos

– NAMT, SIMA, and ATP reviews – Industry conferences: IPC, NEMI, ITEM2000

Collaborate with NEMI, IPC, Georgia Tech, UMBC Evaluating industry specifications (e.g., RosettaNet,

OAG, C-Net, eCo FW) for B2B interactions Evaluating applicability of agent and web

technologies


Continue industry collaborations Demonstrate distributed ICM testbed

operations (with GaTech, NEMI, IPC partners)

Test and validate ICM information model in ICM testbed

Continue evaluation of applicable standards and technologies and their incorporation in ICM testbed

FY00 ResourcesFY00 Resources 1.5 FTE $350K (STRS-EEEL, ATP)

FY01 Resources 2 FTE (other FTE from EEEL, MEL,

MEP) Travel required to collaborate,

participate in standards meetings, and present work

ICMICM

CustomersCustomers– Consumers: PC assembly supply chain participantsConsumers: PC assembly supply chain participants– Consortia: NEMI, IPC, Rosetta NetConsortia: NEMI, IPC, Rosetta Net

ImpactImpact– Reduce the cost of circuit-board manufacturing by Reduce the cost of circuit-board manufacturing by

enabling the rapid exchange of build information and enabling the rapid exchange of build information and change notices for all supply-chain participants change notices for all supply-chain participants

– Demonstrates the use of XML in a vertical industryDemonstrates the use of XML in a vertical industry

StrategyStrategy

Conformance TestsConformance Tests Reference ImplementationsReference Implementations

• Standards developed jointly Standards developed jointly with industrywith industry

Diagnostic TestsDiagnostic Tests Research to reduce costs of testingResearch to reduce costs of testing Reference DataReference Data

Forward-Looking StandardsForward-Looking Standards

Role-Based Access Control (RBAC)Role-Based Access Control (RBAC)

Virtual Reality Modeling Language (VRML)/X3DVirtual Reality Modeling Language (VRML)/X3D

XML and companion standards (DOM, CSS, etc)XML and companion standards (DOM, CSS, etc)

JavaJava

ADLADL

IMSIMS

DASE/SMPTEDASE/SMPTE

OASIS Registry/RepositoryOASIS Registry/Repository

GISGIS

STANDARDS COMMITTEES AND RELATED ACTIVITIES

NATIONAL•NCITS (Previously ANSI accredited X3) Management Level JTC1-TAG Technical Committees H2 (Database) H3 (Computer Graphics) L1 (Geographic Information Systems) L8 (Information Representation) J22 (Java) R1 (Real Time Systems)

IEEE Standards Board 1484

INTERNATIONAL•ISO SC32 (NCITS-L8, NCITS-H2) SC22 (NCITS-J22) SC24 (NCITS-H3) TC211 (NCITS-L1) SC 36 (IEEE 1484)

•ECMA TC39 (JavaScript)

OTHER•World Wide Web Consortium XML, DOM, XSLT, CSS•IMS Global Learning, Inc. •Object Mgmt. Group•OASIS•Web3D Consortium•ebXML (UN/CEFACT)

FEDERAL GOVERNMENT

CIO Council Interoperability Committee Standards Working Group

Real-Time JavaReal-Time Java

Project team:Project team:

Lisa CarnahanLisa CarnahanAlden DimaAlden Dima

Assistance asked by industry providers and government usersAssistance asked by industry providers and government users

Collaborate with Sun, HP, IBM, New Monics, FTLCollaborate with Sun, HP, IBM, New Monics, FTL

Hosted continuing workshops for real-time Java requirementsHosted continuing workshops for real-time Java requirements

Addresses real-time issues in both Java embedded and Java server Addresses real-time issues in both Java embedded and Java server application environmentsapplication environments

NIST coalesced industry to produce one set of requirements leading NIST coalesced industry to produce one set of requirements leading to one specto one spec

Open consensus forum for requirementsOpen consensus forum for requirements

BOTH camps (Sun and HP, Microsoft, etc.) have used NIST-led BOTH camps (Sun and HP, Microsoft, etc.) have used NIST-led requirements document to produce specsrequirements document to produce specs


EE TimesEE Times

“ “Interestingly, both the splinter group Interestingly, both the splinter group and Sun are using as a serious basis for and Sun are using as a serious basis for their efforts a ‘requirements document’ their efforts a ‘requirements document’ for real-time (Java) extensions prepared for real-time (Java) extensions prepared by a separate group working under the by a separate group working under the auspices of the National Institute of auspices of the National Institute of Standards and Technology (NIST). That Standards and Technology (NIST). That is raising hopes that both standards is raising hopes that both standards might ultimately come together.”might ultimately come together.”

Real-time Java Requirements Group

•“Requirements for Real-time Extensions to the Java Platform” (final release -September, 1999)

“The National Institute of Standards and Technology (NIST) has organized a real-time Java requirements group. This is an open group that is defining requirements for real-time Java. Their requirements document will be a valuable input to the experts group,…”

JSR-000001 Real-time Specification for Java (Call for forming the RT Java Experts Group sponsored by Sun Microsystems; NIST member)

Sun Microsystems’ sponsored RT Java Expert Group

“The J Consortium uses real-time requirements developed by the Requirements Group for Real-Time Extensions for the Java Platform, an effort sponsored by the

National Institute of Standards and Technologies (NIST)….”

CUPERTINO, Calif., September 24, 1999[Press Release -J Consortium™ Unveils First Public Real-Time Specification For The Java Platform]

The J Consortium

•NIST’s Requirements Group effort dissolved after successful completion of its goal.


Real-Time Java Requirements document Real-Time Java Requirements document completedcompleted

Worked with both spec development Worked with both spec development groups to help coalesce industrygroups to help coalesce industry

Member of Sun RT Experts Group - Member of Sun RT Experts Group - contributed to specificationcontributed to specification

Worked with J Consortium to help Worked with J Consortium to help develop specificationdevelop specification

Project sunsetProject sunset


FY01 ResourcesFY01 Resources 0 (Project sunset)0 (Project sunset)


CustomersCustomers– Sun, IBM, HP, NewMonics and users of real-Sun, IBM, HP, NewMonics and users of real-

time Javatime Java

ImpactImpact– Coalesced industry to produce one set of Coalesced industry to produce one set of

requirementsrequirements– 2 spec efforts compatible2 spec efforts compatible– New book on Real-time Java mentions NISTNew book on Real-time Java mentions NIST– Presented to and endorsed by VCATPresented to and endorsed by VCAT

ADL ADL (Architecture Description (Architecture Description

Language)Language)

Project Team:Project Team:Chris DabrowskiChris DabrowskiJohn TebbuttJohn Tebbutt

7

ADLs specify (at a minimum)

Use well-defined semantics & syntax

•Lexical and/or graphical language

•Usually use a formal descriptive method to specify aspects (e.g., behavior or constraints)

Support automated analysis & verification of design

•Simulation of execution•abstract formal theory (e.g. posets, fsm)•basis for conformance testing to Ref. Arch.

•Use of ‘formal methods’ approaches to:

•determine internal consistency

•conformance to ‘architectural style rules’

Job AssignorInterface

SchedulerInterface

Plan SelectorInterface

•System structure

•Components & Components Interfaces

•Defined Connections

Job AssignorInterface

Underlying ModuleThat Implements

Behavior

Do_Task(?Job)

Schedule_Job(?Job)

Architecture Description Architecture Description LanguageLanguage

Focus on high-level structure of the Focus on high-level structure of the overall application overall application (i.e., systems architecture)(i.e., systems architecture)

Provide features for modeling software Provide features for modeling software systems architecturesystems architecture– E.g., Structure, components, interfaces, interconnectionsE.g., Structure, components, interfaces, interconnections

Support automated analysis and Support automated analysis and verification of designverification of design

Enable testing during design phase Enable testing during design phase – Saves $ over testing at code phaseSaves $ over testing at code phase

Architecture Description Architecture Description Language (ADL)Language (ADL)

Software Architecture: “..an abstract system specification consisting primarily of functional components described in terms of their behaviors and interfaces and component-component interconnections” [Hayes-Roth, 1994]

ADL: “A language that provides features for modeling a software system’s conceptual architecture” [Medvidovic 1996]

Analysis Design Implement-tation

Testing Maintenance

Focus

Support

Why ADLs?

• Complexity of s/w - formal verification

• Software re-use - strong architecture description

• Testing during design phase - saves $ over testing at code phase

Standards CollaborationsStandards Collaborations– Propose additional functionality to SAE (first industry standards effort)Propose additional functionality to SAE (first industry standards effort)– Uniquely positioned to bring academic and industry communities together.Uniquely positioned to bring academic and industry communities together.– Apply developing AADL standard (based on Honeywell’s MetaH) to Unmanned Apply developing AADL standard (based on Honeywell’s MetaH) to Unmanned

Ground Vehicle: build additional functionality into AADL standardGround Vehicle: build additional functionality into AADL standard– Explore ADL relationships with IEEE and Open Group effortsExplore ADL relationships with IEEE and Open Group efforts

NIST’s Unique Role

Applied Research: Contribute to ADL body-of-knowledgeApplied Research: Contribute to ADL body-of-knowledge– Research application of ‘formal approaches’ to ADLsResearch application of ‘formal approaches’ to ADLs– Enhance ADL features for verifying conformance of application systems architectures to reference architectures (e.g. RCS)Enhance ADL features for verifying conformance of application systems architectures to reference architectures (e.g. RCS)– Publish successes and lessons-learned for industry awarenessPublish successes and lessons-learned for industry awareness– Provide feedback to appropriate ADL tool developersProvide feedback to appropriate ADL tool developers

Standards Role: Neutral Expert

Applied Research: Contribute to ADL body-of-knowledge


Participated in development of SAE’s Avionics ADL (AADL) standard

Explore ADL standardization efforts in IEEE and Open Group

Established partnerships with DISA, US Army, ITL/Security Division

Identify domain areas for applying ADL to explore standardization and testing issues – E.g., government smartcard, unmanned ground vehicle,

pervasive computing

FY01 PlansFY01 Plans Provide support for the formalization of AADL to

ensure its testability Develop prototype of US Army robotics architecture

– collaboration with MEL– dependent on DISA funds

Develop prototype of Smartcard Architecture w. Div 893

Demonstrate and document the benefits and functionality of ADLs

Produce a comparative study of different ADLs to support further development and standardization of ADLs


1 FTE $200K (STRS)

FY01 Resources 2 FTE Equipment and Software: 10K Travel required to standards meetings

and to participate in conferences and industry meetings

Architecture Description Architecture Description LanguageLanguage

CustomersCustomers– Consumers: Industries that rely on component-based Consumers: Industries that rely on component-based

systems (e.g., manufacturing, avionics, real-time)systems (e.g., manufacturing, avionics, real-time)– ADL tool vendorsADL tool vendors

ImpactImpact– Moved ADL from University research to industry Moved ADL from University research to industry

standardstandard– Provide uniform descriptions of the structure and Provide uniform descriptions of the structure and

function of software. function of software. – Increases software quality by reducing errors through Increases software quality by reducing errors through

testing in design phase.testing in design phase.– Increase the producibility, quality, and reusability of Increase the producibility, quality, and reusability of

component-based software systems. component-based software systems.

CIO Council Standards Working GroupCIO Council Standards Working Group

Technology Transfer Act/OMB Circular A-119 direct federal agencies to Technology Transfer Act/OMB Circular A-119 direct federal agencies to use voluntary consensus standards in lieu of government unique use voluntary consensus standards in lieu of government unique standardsstandards

NIST directed to coordinate federal technical standards activities with NIST directed to coordinate federal technical standards activities with private sector technical standards activitiesprivate sector technical standards activities

NIST/ITL initiated effort to create Standards Working Group within Federal NIST/ITL initiated effort to create Standards Working Group within Federal CIO CouncilCIO Council

Standards Working Group approved late last year by the CIO Council Standards Working Group approved late last year by the CIO Council Interoperability CommitteeInteroperability Committee

Standards Working Group to be focal point of discussion, definition, and Standards Working Group to be focal point of discussion, definition, and coordination of federal requirements for voluntary IT standards, and will coordination of federal requirements for voluntary IT standards, and will assess the results of voluntary IT standards in meeting requirementsassess the results of voluntary IT standards in meeting requirements

Standards Working Group chaired by a member of this Division, and will Standards Working Group chaired by a member of this Division, and will be supported by this Division in its efforts to provide a unified voice to the be supported by this Division in its efforts to provide a unified voice to the voluntary IT standards community for Federal IT standards technical voluntary IT standards community for Federal IT standards technical requirementsrequirements

FY00 ProgressFY00 Progress Prototype of web based support software for

collection, evaluation, and status reporting on Federal Government technical requirements for voluntary standards developed and demonstrated.

Electronic Documents conference held featuring speakers from private industry, Federal Government, and voluntary standards organizations.

Meetings of the Standards Working Group held on an as needed basis.

Standards requirements forwarded to appropriate voluntary standards committees as the requirements are approved.

FY01 PlansFY01 Plans Continue development of web based support

software for Standards Working Group. Make software available for possible use by

other Working Groups within the EIEIT structure. Meetings of the Standards Working Group held

on an as needed basis. Standards requirements forwarded to

appropriate voluntary standards committees as the requirements are approved.

Consider holding second Electronic Documents conference.

FY00 ResourcesFY00 Resources 1.5 FTE $325K (STRS, OA-CIO Council)

FY01 Resources 1.5 FTE – Combination of time from

Group Leader and others in Interoperability Group

CIO Council Standards CIO Council Standards Working GroupWorking Group

CustomersCustomers– Other Federal AgenciesOther Federal Agencies

ImpactImpact– Federal agencies input requirements Federal agencies input requirements

to standards groups and utilize to standards groups and utilize voluntary consensus standardsvoluntary consensus standards

– Eliminates duplicative standardsEliminates duplicative standards

StrategyStrategy

Conformance TestsConformance Tests Reference ImplementationsReference Implementations Standards developed jointly with industryStandards developed jointly with industry •Diagnostic TestsDiagnostic Tests Research to reduce costs of testingResearch to reduce costs of testing Reference DataReference Data

Year 2000 Problems - Project Leader: Gary Fisher

Published bulletin and produced Web site• Raised awareness of issues• Proposed methods for managing conversion

• Defined types of tests needed for date/time testing

• Under evaluation by IEEE

Developed test assertions

Issued change notice to FIPS 4-1 on date formats• Recommend use of four-digit format• Discouraged use of two-digit format

Held international symposium June 9-10, 1997• Management and technical issues• Market development of tools and services• Case studies by companies and governments

Developed source code scanner

Participate in Interagency Year 2000 Task Force


Y2K web site completedY2K web site completed Embedded system study completedEmbedded system study completed

– Report sent to KoskinenReport sent to Koskinen– Problems highlighted and impacts Problems highlighted and impacts

reportedreported– Produced new emphasis on embedded Produced new emphasis on embedded

systemssystems Project sunset Project sunset


0.25 FTE0.25 FTE $50K (STRS)$50K (STRS)


Year 2000Year 2000

CustomersCustomers– All businesses that use computersAll businesses that use computers– Consumers who use computersConsumers who use computers

ImpactImpact– Y2K rollover went without incidentY2K rollover went without incident

StrategyStrategy

Conformance TestsConformance Tests Reference ImplementationsReference Implementations Standards developed jointly with industryStandards developed jointly with industry Diagnostic TestsDiagnostic Tests

• Research to reduce costs of testingResearch to reduce costs of testing

Reference DataReference Data

Automatic Generation of Spec Based Tests Automatic Generation of Spec Based Tests Using Formal MethodsUsing Formal Methods


Paul BlackPaul BlackBill Majurski Bill Majurski Jim LyleJim LyleMicheal KooMicheal Koo

Industry NeedIndustry Need•Software testing is enormous cost to industry:Software testing is enormous cost to industry: 30% - 50% of budget30% - 50% of budget

Development may be 1000’s of staff-yearsDevelopment may be 1000’s of staff-yearsTesting cost may be $ millionsTesting cost may be $ millions

• Current methods: mostly hand generated testsCurrent methods: mostly hand generated tests

InefficientInefficientNot rigorous enoughNot rigorous enough

•Software testing: key to reducing security Software testing: key to reducing security incidents:incidents:

““More than 75% of the incidents…are the direct result of …bugs.” More than 75% of the incidents…are the direct result of …bugs.” and “…more real testing - not 'beta testing' in the marketplace - and “…more real testing - not 'beta testing' in the marketplace - will yield programs that are less vulnerable to attack.” will yield programs that are less vulnerable to attack.”

(Business(Business Week, February 28: "Locking Out the Hackers")Week, February 28: "Locking Out the Hackers")

• Conventional Conventional software specificationssoftware specifications are in are in natural language and so lack precisionnatural language and so lack precision

AmbiguitiesAmbiguitiesInconsistenciesInconsistenciesErrorsErrors

• Conventional Conventional software testingsoftware testing problems: problems:Exhaustive testing not possibleExhaustive testing not possibleTest case selection subjectiveTest case selection subjective

Result = costly and time-consuming testingResult = costly and time-consuming testing

ProblemsProblems

Requirements

Specification

SourceCode

EnglishEnglishLanguageLanguage

ProgrammingProgrammingLanguageLanguage

EnglishEnglishLanguageLanguage

HumanProgrammers

SoftwareTests

review &review &inspectinspect

Current Test Development MethodologyCurrent Test Development Methodology

review &review &inspectinspect

Requirements

Specification

SourceCode

ProgrammingProgrammingLanguageLanguage

MathematicalMathematicalLogicLogic

TestGenerator

SoftwareTests

informalinformalproofproof

formalformalproofproof

Advanced Test Development MethodologyAdvanced Test Development Methodology

MathematicalMathematicalLogicLogic

• Method for automatically generating tests from Method for automatically generating tests from formal specificationsformal specifications

• Associated coverage metricAssociated coverage metric

• Publications and presentationsPublications and presentations

ICFEM (98), WSEST(99), HASE (99), DASC (99)ICFEM (98), WSEST(99), HASE (99), DASC (99) GMU, UMBC, College Park, Brisbane, NRLGMU, UMBC, College Park, Brisbane, NRL

• Feasibility collaborations leading to Feasibility collaborations leading to commercializationcommercialization

Argus, a secure computer operating systemArgus, a secure computer operating system Ford Motor Research Lab: powertrain control systemsFord Motor Research Lab: powertrain control systems Northrop Grumman: fighter radar computersNorthrop Grumman: fighter radar computers

AccomplishmentsAccomplishments

FY00 ProgressFY00 Progress Feasibility tests on models from Ford and Feasibility tests on models from Ford and

ArgusArgus Initial formalization of BLAST (Basic Local Initial formalization of BLAST (Basic Local

Alignment Search Tool: identifies similar DNA Alignment Search Tool: identifies similar DNA sequences) Algorithm completed in sequences) Algorithm completed in collaboration with CSTLcollaboration with CSTL

Hosted Workshop on Testing Software Using Hosted Workshop on Testing Software Using Formal MethodsFormal Methods

Hosted Tutorial on Formal MethodsHosted Tutorial on Formal Methods 3 papers published3 papers published 2 papers submitted and accepted2 papers submitted and accepted 4 papers submitted4 papers submitted 9 talks (6 invited)9 talks (6 invited)


Commercialize ATG Method with Ford Commercialize ATG Method with Ford and software company (tbd)and software company (tbd)

Continue feasibility studies with ArgusContinue feasibility studies with Argus Complete BLAST formalizationComplete BLAST formalization


2.5 FTE2.5 FTE $500K (STRS-Competence)$500K (STRS-Competence)

FY01 ResourcesFY01 Resources 2.5 FTE2.5 FTE Equipment: $5KEquipment: $5K Travel required for validation collaborations Travel required for validation collaborations

and presentationsand presentations

Automatic Generation of Automatic Generation of TestsTests

CustomersCustomers– Software developersSoftware developers– Software end users, e.g., web surfersSoftware end users, e.g., web surfers– Security administratorsSecurity administrators– Researchers in software testing methodologiesResearchers in software testing methodologies

ImpactImpact– Reduce the cost of software testing (50% of Reduce the cost of software testing (50% of development costs)development costs)

– Improve software qualityImprove software quality– Reduce security incidents (75% due to bugs)Reduce security incidents (75% due to bugs)– Ford and Argus interested in commercialization of ATG Ford and Argus interested in commercialization of ATG methodologymethodology– Strongly endorsed by VCATStrongly endorsed by VCAT

Software Testing by Statistical Software Testing by Statistical MethodsMethods

(Competence Project)(Competence Project)

Objective: develop new methods for software testing Objective: develop new methods for software testing based on stochastic processes and metrics, in order tobased on stochastic processes and metrics, in order to– Improve the quality of software developmentImprove the quality of software development

– Develop reliability estimates that software correctly adheres to Develop reliability estimates that software correctly adheres to its specification its specification

Joint project Joint project – Software Diagnostics and Conformance Testing Division (897)Software Diagnostics and Conformance Testing Division (897)

– Statistical Engineering division (898)Statistical Engineering division (898) Collaboration with other NIST Labs (MEL, EEEL, CSTL)Collaboration with other NIST Labs (MEL, EEEL, CSTL)


Len GallagherLen GallagherLynne RosenthalLynne Rosenthal

ProblemProblem

•Exhaustive testing not possibleExhaustive testing not possibleCan show presence of errors, not their absenceCan show presence of errors, not their absence

•Degree of correctness not quantifiedDegree of correctness not quantifiedCan derive subjective level of confidence, not Can derive subjective level of confidence, not

quantitative measurequantitative measure

•Test case coverage not scienceTest case coverage not scienceStrategy is intuitive, based on anecdotal Strategy is intuitive, based on anecdotal

evidenceevidence

Conformance Tests

??????conforminconformin

gg

Software Implementation

100% non conforming100% non conforming


Published paper on Coverage Model of Mutation Testing (with Paul Black)

Produced paper on applying statistical methods to component based testing

Produce paper on Test Input Needed to Improve Reliability

Analysis of effort to apply statistical method (partitioning) to new test development


Continue effort to apply statistical method (partitioning) to new test development

Apply statistical method to pervasive computing applications


1 FTE $200K (STRS-Competence, plus $100K

given to ITL/SED)

FY01 Resources 1 FTE 200K competence funding (plus 100K

given to ITL/SED)

Software Testing by Software Testing by Statistical MethodsStatistical Methods

Customers:Customers:– Conformance Test DevelopersConformance Test Developers

– OurselvesOurselves

– Users of conformance testsUsers of conformance tests

Impact:Impact:– Improve our knowledge and skills in designing Improve our knowledge and skills in designing

and developing quality software. and developing quality software.

– Quantitative measures associated with Quantitative measures associated with conformance testing. conformance testing.

Quantum InformationQuantum Information


Paul BlackPaul BlackNeva CarlsonNeva Carlson

Quantum InformationQuantum Information

Goal: to develop information Goal: to develop information representations and computing machines representations and computing machines that take advantage of quantum effectsthat take advantage of quantum effects

Joint effort of physicists and computer Joint effort of physicists and computer scientistsscientists


Invited to develop joint proposal for Invited to develop joint proposal for Nanotechnology Initiative with PhysicsNanotechnology Initiative with Physics

Invited by NIST Physicists to present Invited by NIST Physicists to present Basic Concepts in Theory of ComputationBasic Concepts in Theory of Computation

Developed Quantum Information DemoDeveloped Quantum Information Demo Developed Quantum Information Developed Quantum Information

BibliographyBibliography Attended weekly Physics Quantum Attended weekly Physics Quantum

Information seminarInformation seminar


Develop products for Nanotechnology Develop products for Nanotechnology Quantum Information projectQuantum Information project

Organize Workshop on Computer Organize Workshop on Computer Science in Quantum InformationScience in Quantum Information

Research questions in testing Quantum Research questions in testing Quantum AlgorithmsAlgorithms

FY00 ResourcesFY00 Resources .5 FTE.5 FTE $100K (STRS – Physics $100K (STRS – Physics

Lab?)Lab?)

FY01 ResourcesFY01 Resources 1.5 FTE (depends on 1.5 FTE (depends on

Nanotechnology Initiative Nanotechnology Initiative or $ from Physics)or $ from Physics)

Equipment: $0Equipment: $0 Travel anticipatedTravel anticipated

Quantum InformationQuantum Information CustomersCustomers

– Researchers in computational theory, and Researchers in computational theory, and algorithm design and algorithm design and analysisanalysis

– Developers of large computer models, e.g., Developers of large computer models, e.g., global weatherglobal weather– CryptanalistsCryptanalists

ImpactImpact– Computing machines of unprecedented Computing machines of unprecedented capability and small sizecapability and small size

– Solutions to intractable computational Solutions to intractable computational problems, e.g., problems, e.g., factoring, searchingfactoring, searching

– ITL staff invited to present at Physics seminars and ITL staff invited to present at Physics seminars and participate in participate in Physics Lab effortsPhysics Lab efforts

StrategyStrategy

Conformance TestsConformance Tests Reference ImplementationsReference Implementations Standards developed jointly with industryStandards developed jointly with industry Diagnostic TestsDiagnostic Tests Research to reduce costs of testingResearch to reduce costs of testing

• Reference DataReference Data

Error, Fault, and Failure (EFF)Error, Fault, and Failure (EFF)Data Collection and Analysis Data Collection and Analysis


Dolores WallaceDolores WallaceMichael KooMichael KooTony CincottaTony CincottaJason RagerJason Rager

Error, Fault, and Failure (EFF)Error, Fault, and Failure (EFF)Data Collection and Analysis Data Collection and Analysis

• Repositories of reference data on Repositories of reference data on software failuressoftware failures

• Tools to create and manage repositoriesTools to create and manage repositories

• Tools to analyze software failure dataTools to analyze software failure data

• Research in taxonomies to characterize Research in taxonomies to characterize software failuressoftware failures

FY00 ProgressFY00 Progress Repository systems:Repository systems:

– – Completed public repository with predefined Completed public repository with predefined queries queries

– – Completed internal repository with dynamic Completed internal repository with dynamic queries queries

Data: 1000 data points addedData: 1000 data points added

Paper on failures in medical devices Paper on failures in medical devices finalizedfinalized

Papers: report on NASA data submitted Papers: report on NASA data submitted to to NASANASA

Project was sunsetProject was sunset

FY00 ResourcesFY00 Resources 1.5 FTE1.5 FTE $300K (STRS, ATP - $160K)$300K (STRS, ATP - $160K)


Error, Fault, and FailuresError, Fault, and Failures CustomersCustomers

– Software DevelopersSoftware Developers– Researchers in Software Development MethodologiesResearchers in Software Development Methodologies

ImpactImpact– Identifies the stages in the software life cycle whose practices Identifies the stages in the software life cycle whose practices

can be improvedcan be improved– Endorsed by the Software Division of the Endorsed by the Software Division of the

American Society for QualityAmerican Society for Quality– Project methodologies and tools used in Project methodologies and tools used in

graduate programs at Johns Hopkins and the graduate programs at Johns Hopkins and the University of LouisvilleUniversity of Louisville

– EFF Tool used by RST CorporationEFF Tool used by RST Corporation

Other Agency ProjectsOther Agency Projects NIJ - Software and Tool VerificationNIJ - Software and Tool Verification

– Project Team: Gary Fisher, Len Gebase, Jim Lyle, Project Team: Gary Fisher, Len Gebase, Jim Lyle, Jackie Schneider, Mark ZimmermanJackie Schneider, Mark Zimmerman

VA - Health Care Information SystemsVA - Health Care Information Systems– Project Leader: Bill MajurskiProject Leader: Bill Majurski

GSA - EC Standards RoadmapGSA - EC Standards Roadmap– Project Leader: Mary BradyProject Leader: Mary Brady

EPA - Metadata Descriptions and EPA - Metadata Descriptions and RegistriesRegistries– Project Leader: Judy NewtonProject Leader: Judy Newton

NIJNIJNational Software Reference Library & Beta National Software Reference Library & Beta

TestingTesting

National Software Reference LibraryNational Software Reference Library GOAL: Reference Data Set of file signatures (hash) of GOAL: Reference Data Set of file signatures (hash) of

COTS fileCOTS file Used during examination of digital evidence to identify Used during examination of digital evidence to identify

pertinent files and eliminate others pertinent files and eliminate others BOTTOM LINE: One computer may have 5,000-20,000 BOTTOM LINE: One computer may have 5,000-20,000

files. NSRL data set can eliminate 40-95% of those files files. NSRL data set can eliminate 40-95% of those files and save hundreds of staff-hours.and save hundreds of staff-hours.

Beta Testing ProjectBeta Testing Project GOAL: Verification of computer forensic tools and their GOAL: Verification of computer forensic tools and their

use in analyzing digital evidenceuse in analyzing digital evidence Used to ensure tool usage yields objective, repeatable, Used to ensure tool usage yields objective, repeatable,

reproducible, results that hold up in court.reproducible, results that hold up in court.


Developed NSRL data dictionary and database structure for hash codes

Established contacts with Microsoft, Adobe, et al to supply software– CBD announcement, Press release

Identified hash algorithms Creating Reference Data Set (i.e, file signatures)

Developed draft test plan for beta testing Develop procedures and tests for imaging tools

FY01 PlansFY01 Plans Establish the reference data set and

distribution mechanism – (SQL database with web interface)

Develop test metrics for efficacy of hash codes used in the reference data set

Test the NSRL web site and make it available to approved users

Continue to develop tests and procedures for forensic tools

Issue test reports and test suites (publish on web site)


3.5 FTE3.5 FTE $700K (OA-NIJ)$700K (OA-NIJ)

FY01 ResourcesFY01 Resources 3.5 FTE3.5 FTE Travel required for training courses Travel required for training courses

and presentationsand presentations

NIJNIJ

Customers:Customers:– Law Enforcement Agencies (e.g., FBI, Treasury, Law Enforcement Agencies (e.g., FBI, Treasury,

DCFL, state and local agencies)DCFL, state and local agencies)

– Software companies (e.g., Adobe, Microsoft)Software companies (e.g., Adobe, Microsoft)

Impact:Impact:– Improve the efficiency of forensic examination Improve the efficiency of forensic examination

of suspect computer files/date (NSRL data set of suspect computer files/date (NSRL data set can eliminate 40-95% of suspect files and save can eliminate 40-95% of suspect files and save hundreds of staff-hours).hundreds of staff-hours).

– Provide testing procedures for forensic Provide testing procedures for forensic software tools to ensure the analysis process software tools to ensure the analysis process and results holds up in courtand results holds up in court

Healthcare Information SystemsHealthcare Information Systems


Bill Majurski Bill Majurski Tony CincottaTony CincottaRoger GimaRoger GimaJohn BarkleyJohn Barkley

GSA - EC Standards GSA - EC Standards RoadmapRoadmap

EPA - Metadata RegistriesEPA - Metadata Registries

Industry TestimonialsIndustry Testimonials ““The OASIS-NIST XML Conformance Test Suite is critical for our The OASIS-NIST XML Conformance Test Suite is critical for our

industry.”industry.”– Norbert Mikula, CTO, OASISNorbert Mikula, CTO, OASIS

““NIST made strong contributions (X3D) and resolved our knottiest NIST made strong contributions (X3D) and resolved our knottiest problems.”problems.”

– Don Brutzman, Board of Dir., Web3D ConsortiumDon Brutzman, Board of Dir., Web3D Consortium ““Your expertise helped advance conformance and spec Your expertise helped advance conformance and spec

development.”development.”– Steve Griffin, IMS Technical DirectorSteve Griffin, IMS Technical Director

““The high quality test suite and certification program is an The high quality test suite and certification program is an invaluable resource for ATA.”invaluable resource for ATA.”

– Robert Peel, Director of Airworthiness and StandardsRobert Peel, Director of Airworthiness and Standards ““NIST’s participation on DASE is vital for the U.S. industry.”NIST’s participation on DASE is vital for the U.S. industry.”

– Lisa McKnight, Multimedia Business DevelopmentLisa McKnight, Multimedia Business Development

NIST Requested Division NIST Requested Division PresentationsPresentations

eCommerce - XML and DASEeCommerce - XML and DASE– Presented to Undersecretary ShaversPresented to Undersecretary Shavers– Presented to NIST Director and DeputyPresented to NIST Director and Deputy

VRML, Real-Time Java, Auto GenerationVRML, Real-Time Java, Auto Generation– Presented to VCATPresented to VCAT

XML/DOM Conformance TestingXML/DOM Conformance Testing– Presented at ITL Town MeetingPresented at ITL Town Meeting

VRML Conformance TestingVRML Conformance Testing– Presented at ITL Town MetingPresented at ITL Town Meting

VmViewVmView– Presented to Congressional staffPresented to Congressional staff

ICM Project BriefingICM Project Briefing– Presented at ATP National MeetingPresented at ATP National Meeting

NIST TestimonialsNIST Testimonials ““Software testing work . . . is highly appropriate serving a Software testing work . . . is highly appropriate serving a

broad set of companies who would not, on their own make broad set of companies who would not, on their own make these investments.”these investments.”

– VCAT ReportVCAT Report ““Way to go folks. You did us proud.”Way to go folks. You did us proud.”

– Brian Belanger, Exec. Director, VCATBrian Belanger, Exec. Director, VCAT ““The model used in the software testing program - to partner The model used in the software testing program - to partner

with industry, rather than developing tests and transferring with industry, rather than developing tests and transferring them is excellent and will become a model for other them is excellent and will become a model for other programs at NIST.”programs at NIST.”

– Ray Kammer’s feedback of the ITL Program ReviewRay Kammer’s feedback of the ITL Program Review ““A wonderful testimonial to your work.”A wonderful testimonial to your work.”

– Ray Kammer’s response to the ATA letter.Ray Kammer’s response to the ATA letter.

PrioritiesPrioritiesCore ProgramCore Program

1.1. XML Conformance testingXML Conformance testing

2.2. Interactive TVInteractive TV

3.3. Automatic Generation of TestsAutomatic Generation of Tests

4.4. Pervasive ComputingPervasive Computing

5.5. XML Registry/RepositoryXML Registry/Repository

----------------------------------------------------------------------------------------------------------------------

6.6. Computer Forensics – NIJComputer Forensics – NIJ

7.7. CIO Council Standards Working GroupCIO Council Standards Working Group

Priorities – Next LevelPriorities – Next Level

1.1. ADLADL

2.2. Health Care Information Systems – VAHealth Care Information Systems – VA

3.3. CGMCGM

4.4. Learning TechnologiesLearning Technologies

5.5. Internet Commerce for ManufacturingInternet Commerce for Manufacturing

6.6. Software Testing by Statistical MethodsSoftware Testing by Statistical Methods

7.7. Quantum InformationQuantum Information

8.8. Other OA Projects (GSA, EPA)Other OA Projects (GSA, EPA)

Division IssuesDivision Issues We’re a victim of our success - we’re in We’re a victim of our success - we’re in

constant demandconstant demand We need people!!!!!We need people!!!!! We need more discretionary fundingWe need more discretionary funding

– STRS vs. OA/ATPSTRS vs. OA/ATP Travel demands are increasingTravel demands are increasing Demonstrations - good and badDemonstrations - good and bad Membership in consortia is essentialMembership in consortia is essential Collaborations with other DivisionsCollaborations with other Divisions

ROSENTHAL ROSEN

SKALL

BARKLEY

We also consult with distinguished scientistsWe also consult with distinguished scientists

Remember, Al and the SDCT Division jointly invented the Internet.Remember, Al and the SDCT Division jointly invented the Internet.

Software Testing Conformance Diagnostics And Mark W. Skall, Division Chief Candy Leatherman,...

Documents

Transcript of Software Testing Conformance Diagnostics And Mark W. Skall, Division Chief Candy Leatherman,...