IT STANDARDS TUTORIAL Part IV – Standardization Processes

11

ITITSTANDARDSSTANDARDSTUTORIALTUTORIAL

Part IV – Part IV – Standardization Standardization

ProcessesProcesses

22

Goals of Standards ProcessGoals of Standards Process

• Well-Defined Product:Well-Defined Product:– Consistent implementationsConsistent implementations– Coherent functionalityCoherent functionality

• Commercial Viability:Commercial Viability:– Allows range of implementationsAllows range of implementations– Commercial products are possibleCommercial products are possible– Promotes wide adoptionPromotes wide adoption– No “Standards-for-Standards-Sake” (e.g., some No “Standards-for-Standards-Sake” (e.g., some

standards consultant dominated projects)standards consultant dominated projects)• Wide acceptance:Wide acceptance:

– Many conforming implementationsMany conforming implementations• Few bugs:Few bugs:

– Low number of defect reportsLow number of defect reports

33

IT Standards Development IT Standards Development ProcessesProcesses• International Standards DevelopmentInternational Standards Development

• De jure ProcessDe jure Process

• Professional Society ProcessProfessional Society Process

• Industry Association ProcessIndustry Association Process

• Consortia ProcessConsortia Process

• Government ProcessGovernment Process

44

DIFFERENT APPROACHESDIFFERENT APPROACHES

TOP DOWNTOP DOWN - Treaty-type organization such as the ITU

BOTTOM-UPBOTTOM-UP - Voluntary-type organization such as ISO/IEC

“FLY-BEFORE-YOU-BUYFLY-BEFORE-YOU-BUY” - Internet Society approach

55

TYPESTYPES

1 - OPEN- OPEN

Standards and specifications are considered “open” when sponsored and supported by an organization which uses an open, public consensus process to develop and maintain them

2 - CLOSEDCLOSED

Controlled, usually, in a proprietary fashion; often referred to as de facto specifications

66

TYPESTYPES

TWO KINDS OF OPEN STANDARDS:TWO KINDS OF OPEN STANDARDS:

* Formally recognizedFormally recognized standards body which produces and distributes formal or de jure, public standards

* Informal organizationsInformal organizations that produce specifications such as industry standards; called de facto specifications (which are sometimes submitted to formal standards bodies for accreditation or adoption to become formally recognized)

77

CATEGORIESCATEGORIES

1 - 1 - PRODUCTPRODUCT Standards Standards2 - 2 - CONTROLCONTROL Standards Standards3 - 3 - PROCESSPROCESS Standards Standards4 - 4 - PERFORMANCEPERFORMANCE Standards Standards

88


PRODUCT STANDARDSPRODUCT STANDARDS

Product standards embody information. They specify the characteristics of a product

and permit product identification, interoperability, and quality control.

E.g., VCR standards, machine bolt thread sizes, encryption specifications, etc.

99


CONTROL STANDARDSCONTROL STANDARDS

Control Standards are designed to address a societal hazard or problem. They generally define a range of acceptability with respect to the design, performance,

and/or use of a product. They often appear in theform of regulations e.g., building codes, fuel economy

standards, auto safety, etc.

1010

CATEGORIESCATEGORIESPROCESS STANDARDSPROCESS STANDARDS

Process Standards facilitate and support socioeconomic transactions and interactions. They define roles and

relationships, establish the rules of interpreting behavior, and specify the way in which a particular procedure or process is executed. Examples are EDI specifications

that not only govern standardized electronic procedures for economic interactions, but also establish common protocols for business interactions and determine roles

and relationships between suppliers, manufacturers, and customs. E.g., language, customs, bills of lading,

growing of Smithfield ham, certified processes, etc.

1111


PERFORMANCE STANDARDSPERFORMANCE STANDARDS

Performance standards state requirements in terms of required results with criteria for verifying

compliance, but without stating the methods [processes] for achieving the required results. They

define the ‘functional’ requirements for the item, environment in which it must operate, and interface and interchangeability characteristics. E.g., SGML,

CGM, STEP, etc.

1212

““12 Ways12 Ways

toto

PublishPublish

YourYour

SpecificatioSpecificatio

n”n”

1313

What Do You Want To BeWhat Do You Want To BeWhen You Grow Up?When You Grow Up?

ISOISO

StandardStandard

ISOISO

TechTech

SpecSpec

ConsortiaConsortia

SpecSpec

IndustryIndustry

SpecSpec

Professional

Professional

SocietySociety

StandardStandard

de jurede jure

StandardStandard

PubliclyPublicly

AvailableAvailable

SpecSpec

Multi-Multi-

LateralLateral

AgreementAgreement

NationalNational

StandardStandard

1414

How Mature Is Your Base?How Mature Is Your Base?

• Technology Currency?Technology Currency?

• Global Participation?Global Participation?

• Stability?Stability?

• Ubiquity?Ubiquity?

• Vendor Support?Vendor Support?

• Implementation Extent?Implementation Extent?

1515

12 WAYS...12 WAYS...

•3 Kinds of Sponsors:3 Kinds of Sponsors:A) Accredited Standards BodyA) Accredited Standards Body

B) GovernmentB) Government

C) Industry Association / C) Industry Association / Professional Society / Professional Society /

ConsortiumConsortium

1616

12 WAYS...12 WAYS...

• GovernmentGovernment 1.1. National Standard (each National Body)National Standard (each National Body)2.2. Multi-Lateral AgreementMulti-Lateral Agreement

• Non-Accredited OrganizationNon-Accredited Organization3.3. Professional Society/Industry Specification Professional Society/Industry Specification 4.4. ConsortiaConsortia

• De jure Accredited Standards De jure Accredited Standards BodyBody5.5. SC4 New ProjectSC4 New Project6,7,8. SC4 ‘External Harvesting’ Alternatives6,7,8. SC4 ‘External Harvesting’ Alternatives

9.9. ISO/IEC JTC1 SC32ISO/IEC JTC1 SC3210.10. JTC1 PASJTC1 PAS11.11. Other ISO or IEC TCOther ISO or IEC TC12.12. New Direct ProcessNew Direct Process

1717

INTERNATIONALINTERNATIONALSTANDARDSSTANDARDS

DEVELOPMENTDEVELOPMENT

1818

New WorkItem

New WorkItem

Generic Steps to Developing anInternational Standard

CommitteeDraft

CommitteeDraft

Draft forPublic Review

Draft forPublic Review

InternationalStandard


STAGESSTAGESOFOF


STAGESSTAGESOFOF


1919

New WorkItem

New WorkItem

NWINWINWINWI

Steps to Becoming anAccredited InternationalStandard (IS) in ISO

CommitteeDraft

CommitteeDraft

DraftInternational

Standard

DraftInternational

Standard



ISIS

DISDIS

CDCD

STAGESSTAGESOFOF


STAGESSTAGESOFOF

DEVELOPMENTDEVELOPMENT2 - 3 years

2020

WorkingGroup

WorkingGroup

SubCommittee

SubCommittee

TechnicalCommittee

TechnicalCommittee

ISOISO

ORGANIZATIONORGANIZATIONHEIRARCHYHEIRARCHY

Steps to Becoming anAccredited InternationalStandard (IS)

2121

Technical EditorTechnical Editor

WorkingGroup

WorkingGroup

SubCommittee

SubCommittee

TechnicalCommittee

TechnicalCommittee

ISOISO

National Body Technical ExpertsNational Body Technical Experts



2222

Steps to Becoming anAccredited Standard



TechnicalEditor

WorkingGroup Convenor

National Body Technical Experts

2323



TechnicalEditor

WorkingGroup Convenor

National Body Technical Experts

U.S.U.S.

MicroMicroSoftSoft HPHP SUNSUN IBMIBM DODDODNCITSNCITSIEEEIEEEEtcEtc CompacCompac

U.S.U.S.ParticipationParticipation

US ExampleUS Example

2424

Technical EditorTechnical Editor

WorkingGroup 8

WorkingGroup 8

SC24

SC24

JTC1JTC1

ISOISO

National Body Technical ExpertsNational Body Technical Experts

CGMExample


2525

Dra

ftD

raft

Sp

ecifi

cati

on

Sp

ecifi

cati

on

TechnicalEditors

TechnicalEditors

SC24

SC24

JTC1JTC1

ISOISO

VRML CONSORTIUMVRML CONSORTIUM

VRMLExample


2626


• An engineering and management processAn engineering and management process

• Consider relevant industry input: Need to Consider relevant industry input: Need to include all stakeholders with “material include all stakeholders with “material interest”, e.g., vendors, testers, interest”, e.g., vendors, testers, producers, users, government, etc.producers, users, government, etc.

• Produces a technical document Produces a technical document (specification)(specification)

2727

DEVELOPMENT DEVELOPMENT Engineering Process Engineering Process [1][1]

• Identify scopeIdentify scope

• Use proven engineering methodologyUse proven engineering methodology

• Preference for existing commercial Preference for existing commercial practicepractice

• Don’t standardize implementations - Don’t standardize implementations - Standardize specificationsStandardize specifications

2828

DEVELOPMENTDEVELOPMENTEngineering Process Engineering Process [2][2]

• Can new technology be Can new technology be incorporated?incorporated?

• Technology horizon:Technology horizon:– Which technology incorporated? Which technology incorporated?

Feasible/commercial, up to 1-2 years Feasible/commercial, up to 1-2 years from nowfrom now

– How long should standard be useful? At How long should standard be useful? At least 5-10 years from nowleast 5-10 years from now

2929


• Risk management: scope, prior art, Risk management: scope, prior art, scheduleschedule

• Determining requirementsDetermining requirements

• Asking the right questions (1-3 years?)Asking the right questions (1-3 years?)

• Document organization and phasingDocument organization and phasing

• Base documents -- starting pointsBase documents -- starting points

• Proposals and papers -- additionsProposals and papers -- additions

3030


• Developing standards wording ...Developing standards wording ...– Step #0: Requirements identification (optional)Step #0: Requirements identification (optional)– Step #1: Functionality: what it doesStep #1: Functionality: what it does– Step #2: Conceptual model: how it worksStep #2: Conceptual model: how it works– Step #3: Semantics: precise description of Step #3: Semantics: precise description of

featuresfeatures– Step #4: Bindings: transform into target, e.g., Step #4: Bindings: transform into target, e.g.,

application programming interfaces (APIs), syntax, application programming interfaces (APIs), syntax, commands, protocol definition, file format, codingscommands, protocol definition, file format, codings

– Step #5: Encodings: bit/octet formats (optional)Step #5: Encodings: bit/octet formats (optional)– Step #6: Standards words: “legal” wordingStep #6: Standards words: “legal” wording

3131


• Let bake – settle over timeLet bake – settle over time– Baking is very importantBaking is very important– Shakes out subtle bugsShakes out subtle bugs– Greatly improves quality!Greatly improves quality!– Usually, vendors fight “baking” ... want Usually, vendors fight “baking” ... want

to announce “conforming” products to announce “conforming” products ASAPASAP

3232

DEVELOPMENTDEVELOPMENTConsider Relevant InputConsider Relevant Input

• Stakeholders/Sources:Stakeholders/Sources:– Commercial products and servicesCommercial products and services– Industry participantsIndustry participants– GovernmentGovernment– DevelopersDevelopers– ProducersProducers– ManufacturersManufacturers– TestersTesters– UsersUsers– ResearchersResearchers– Etc.Etc.

3333

DEVELOPMENTDEVELOPMENTTechnical Specification Technical Specification [1][1]

• Assertions:Assertions:– Sentences using “shall”Sentences using “shall”– Weaker assertions: “should”, “may”Weaker assertions: “should”, “may”

• Inquiries/Ranges:Inquiries/Ranges:– Implementations have varying limitationsImplementations have varying limitations– Interoperability: tolerances vs. minimumsInteroperability: tolerances vs. minimums– Allows implementation-defined and Allows implementation-defined and

unspecified behaviorunspecified behavior

• Negotiations:Negotiations:– Adaptation to conformance levelsAdaptation to conformance levels

3434


•Conformance:Conformance:– Complies with all assertionsComplies with all assertions– Performs in range allowed Performs in range allowed

within inquiries/ranges and within inquiries/ranges and negotiationsnegotiations

– Minimize implementation-Minimize implementation-defined, unspecified, and defined, unspecified, and undefined behaviorundefined behavior

3535


• Applications and standards use:Applications and standards use:– Strictly conforming: uses features Strictly conforming: uses features

that exist in all implementationsthat exist in all implementations– Conforming: uses features in some Conforming: uses features in some

conforming implementationsconforming implementations– Conformance levels vs. Conformance levels vs.

interoperabilityinteroperability

3636


• Rationale:Rationale:– Explanation of important committee Explanation of important committee

decisionsdecisions– Features considered/rejectedFeatures considered/rejected– Allows for change in membershipAllows for change in membership– Helps with Requests for Helps with Requests for

InterpretationInterpretation

3737

Building ConsensusBuilding Consensus• Public review cycles:Public review cycles:

– Related industries and stakeholdersRelated industries and stakeholders– LiaisonsLiaisons– Responding to commentsResponding to comments

• Ballot:Ballot:– Solidifying consensus -- issues resolved, Solidifying consensus -- issues resolved,

not revisitednot revisited– Resolving disputesResolving disputes– May require several iterationsMay require several iterations

• Approval:Approval:– Document can be publishedDocument can be published– Can measure conformanceCan measure conformance

3838

Maintenance Maintenance [1][1]

• Defect reports:Defect reports:

– Bug reportsBug reports

– Fix features, as necessaryFix features, as necessary

– Many defect reports ==> weak standardMany defect reports ==> weak standard

– Weak standard ==> little acceptance/useWeak standard ==> little acceptance/use

– Little acceptance/use ==> failureLittle acceptance/use ==> failure

– Consistent responses -- highly desirableConsistent responses -- highly desirable

3939

Maintenance Maintenance [2][2]

• Other deliverables:Other deliverables:

– Record of responses (to RFIs)Record of responses (to RFIs)

– Technical corrigenda (for defect reports)Technical corrigenda (for defect reports)

– AmendmentsAmendments

• Next revision:Next revision:

– ReaffirmReaffirm

– ReviseRevise

– WithdrawWithdraw

4040

STANDARDSSTANDARDSTUTORIALTUTORIAL

Part V – Standards Part V – Standards Life CycleLife Cycle

4141

Comparison of Life Comparison of Life CyclesCycles

Highlight similaritiesIEEE, W3C, IETF, de jure, ITU, etc.

4242

Standards DevelopmentStandards Development

User / Vendor Needs

AssessmentInternational Standard (IS)

Working Draft (WD)

Development

Committee Draft (CD)

Consensus and

Verification

Draft International

Standard (DIS)

Validation, Consensus and Fixes

4343

Paradigm ComparisonsParadigm Comparisons

• De jure community (ISO, ANSI, NCITS, etc.)De jure community (ISO, ANSI, NCITS, etc.)

• Consortia (OASIS, W3C, IETF)Consortia (OASIS, W3C, IETF)

• Professional Society (IEEE, ACM)Professional Society (IEEE, ACM)

• Industry Association (EIA/GEIA, etc.)Industry Association (EIA/GEIA, etc.)

• Federal Government (DSP, FIPS, FGDC)Federal Government (DSP, FIPS, FGDC)

4444

Paradigm ComparisonsParadigm ComparisonsDe Jure De Jure

CommunitCommunityy

ConsortiaConsortia ProfessionProfessional Societyal Society

Industry Industry AssociationAssociation

Federal Federal GovernmeGovernme

ntnt

Sample Sample OrganizationsOrganizations

ISO, ANSI, ISO, ANSI, NCITSNCITS

OASIS, W3C, OASIS, W3C, IETFIETF IEEE, ACMIEEE, ACM EIA/GEIA/TIAEIA/GEIA/TIA DSP, FIPS, DSP, FIPS,

FGDCFGDC

New Work New Work ItemsItems

Proposal Proposal approved by approved by at least 5 at least 5 National National BodiesBodies

At least 3 At least 3 members members draft charter. draft charter. Approval by Approval by OASIS TC OASIS TC AdministratioAdministration.n.

Establish Establish sponsorship sponsorship under IEEE under IEEE Society, TC, or Society, TC, or SCC. Must SCC. Must have member have member of IEEE-SA of IEEE-SA Board on Board on committee.committee.

Formulating Group Formulating Group submits proposal to submits proposal to Technical Standards Technical Standards Subcommittee for Subcommittee for Approval. Referred to Approval. Referred to an Engineering an Engineering Committee for Committee for development.development.

FIPS FIPS Standards Standards developed by developed by NIST. FGDC NIST. FGDC standards standards developed by developed by working working groups within groups within FGDC FGDC Committees.Committees.

Technical Technical MembershipMembership

Appointed by Appointed by National National BodiesBodies

““Eligible” Eligible” individual individual members members volunteer.volunteer.

Appointed by Appointed by IEEE Society IEEE Society or IEEE-SA or IEEE-SA Board. IEEE Board. IEEE Members can Members can apply for apply for membership membership to the Chair.to the Chair.

Member companies Member companies appoint voting appoint voting representative. May representative. May designate designate supplemental supplemental representatives. representatives. Non-TIA companies Non-TIA companies may pay a fee to may pay a fee to participate.participate.

Member of Member of FGDC FGDC workgroup or workgroup or a NIST a NIST employee/ employee/ contractor.contractor.

Time Limit to Time Limit to CompleteComplete

36-48 Months36-48 Months

Completion Completion dates dates established in established in the TC the TC Charter.Charter.

4 Years4 Years

Submitted for Submitted for publication within 1 publication within 1 year from the close of year from the close of the comment period.the comment period.

None StatedNone Stated

4545


CommunityCommunity ConsortiaConsortia Professional Professional SocietySociety


Federal Federal GovernmentGovernment

Sample Sample OrganizatioOrganizationsns



FGDCFGDC

Vote on Vote on StandardStandard

2/3 majority 2/3 majority of the votes of the votes cast, not cast, not more than more than 1/4 negative.1/4 negative.

2/3 majority 2/3 majority for Committee for Committee Draft, For Draft, For OASIS OASIS Standard, TC Standard, TC majority vote majority vote to submit for to submit for public review, public review, Certification Certification by at least by at least three OASIS three OASIS member member organizations organizations that they are that they are successfully successfully using the using the Specification, Specification, 15% approval 15% approval of entire of entire OASIS OASIS organizational organizational members.members.

Balloting Balloting Group formed. Group formed. Must have Must have 75% return of 75% return of votes. votes. Minimum of Minimum of 75% of those 75% of those voting voting required for required for approval to approval to submit submit standard to standard to IEEE-SA IEEE-SA Board.Board.

Majority vote Majority vote of IEEE-SA of IEEE-SA Standards Standards Board Board required for required for IEEE IEEE Standard.Standard.

Voting must reach Voting must reach consensus. consensus. Consensus – Consensus – established when established when those participating in those participating in the vote have the vote have reached substantial reached substantial agreement. agreement. Substantial Substantial agreement means agreement means more than a simple more than a simple majority, but not majority, but not necessarily unanimity. necessarily unanimity. Consensus requires Consensus requires that all views and that all views and objections be objections be considered and that a considered and that a concerted effort be concerted effort be made toward their made toward their resolution. Consensus resolution. Consensus may also be achieved may also be achieved when the minority no when the minority no longer wishes to longer wishes to articulate its articulate its objection. objection.

FIPS and FIPS and FGDC FGDC Standards are Standards are published in published in the Federal the Federal Register for Register for public public comment. comment.

FIPS FIPS standards standards approved by approved by Secretary of Secretary of Commerce. Commerce.

FGDC FGDC Standards Standards approved by approved by FGDC FGDC Steering Steering CommitteeCommittee

4646


CommuniCommunityty

ConsortiaConsortia ProfessionaProfessional Societyl Society


Federal Federal GovernmenGovernmen

tt

Sample Sample OrganizationsOrganizations



FGDCFGDC

Review of Review of existing existing StandardsStandards

Every 5 Every 5 YearsYears None StatedNone Stated Every 5 YearsEvery 5 Years

Every 5 Years if Every 5 Years if published as an ANSI published as an ANSI

StandardStandardNone StatedNone Stated

DistributionDistribution

Sold thru ISO Sold thru ISO and National and National Standards Standards BodiesBodies

OASIS OASIS Standards Standards available on available on the WEB at no the WEB at no ChargeCharge

Available for Available for sale from IEEE sale from IEEE or IEC/ANSI if or IEC/ANSI if joint joint IEEE/IEC/ANSI IEEE/IEC/ANSI standardstandard

Sold through TIA Sold through TIA and/or ANSI if joint and/or ANSI if joint standardstandard

FGDC and FGDC and some FIPS some FIPS Standards Standards available on available on WEB at no WEB at no chargecharge

4747


CommunityCommunityConsortiConsorti

aaProfessional Professional

SocietySocietyIndustry Industry

AssociationAssociation

Federal Federal GovernmenGovernmen

tt

Sample Sample OrganizatioOrganizationsns

ISO, ANSI, NCITSISO, ANSI, NCITS OASIS, OASIS, W3C, IETFW3C, IETF IEEE, ACMIEEE, ACM EIA/GEIA/TIAEIA/GEIA/TIA DSP, FIPS, DSP, FIPS,

FGDCFGDC

IPR IssuesIPR Issues

Copyright for all Copyright for all drafts and drafts and International International Standards and Standards and other other publications publications belongs to ISO belongs to ISO or IECor IEC

IPR rights IPR rights transferred transferred to OASIS to OASIS for free for free distribution distribution of of materialsmaterials

All IEEE All IEEE standards are standards are copyrighted by copyrighted by the IEEE the IEEE

Copyright for all drafts Copyright for all drafts and TIA Standards and TIA Standards and other publications and other publications belongs to TIA. Joint belongs to TIA. Joint Standards may be Standards may be copyrighted by all copyrighted by all pertinent Standards pertinent Standards Development Development Organizations Organizations involved in involved in development as development as determined by determined by agreement among agreement among them.them.

No copyrights No copyrights for material for material developed by developed by U.S. U.S. Government Government funding.funding.

AccreditatioAccreditation n

ISO recognizes ISO recognizes ANSI as the US ANSI as the US Standards Body.Standards Body.

Not Not accreditedaccredited

Accredited by Accredited by ANSIANSI Accredited by ANSIAccredited by ANSI N/AN/A

Note: ANSI accreditation implies that accredited organizations will comply with ANSI rules and processes for developing standards that will be submitted for ANSI or International Standard approval.

4848

Generic IT Standards Life Generic IT Standards Life CycleCycle

Development ConsensusBuilding

Maintenance

Revise, Reaffirm, Withdraw

• Choosing the right “process” is not trivial

• Accreditation affords consistent process

• Committees don’t reinvent wheel

• Accredited process is well-tested and “off the shelf”

• Consensus is significant

• Broad participation yields better quality results but make for slower process

Consistency Via Accredited ProcessConsistency Via Accredited Process

4949

Generic IT Standards Life Generic IT Standards Life Cycle:Cycle:

DEVELOPMENTDEVELOPMENT• Source: “from scratch” Source: “from scratch”

or “base documents”or “base documents”

• Create “standards Create “standards wording” (normative wording” (normative and informative), and informative), rationale for decisionsrationale for decisions

• Technical committee: Technical committee: in-person or electronic in-person or electronic collaborationcollaboration

DEVELOPMENT

5050


CONSENSUS BUILDINGCONSENSUS BUILDING• Collaboration, harmon-Collaboration, harmon-

ization, refinementization, refinement• Public reviews as soon Public reviews as soon

as possibleas possible• Public commentsPublic comments• Resolution of commentsResolution of comments• Approval stages:Approval stages:

– Working draftWorking draft– Committee draftCommittee draft– Draft StandardDraft Standard– Approved StandardApproved Standard

Development CONSENSUSCONSENSUSBUILDINGBUILDING

5151


MAINTENANCEMAINTENANCE• Requests for Requests for

Interpretation Interpretation (RFIs)(RFIs)

• Defect Reports Defect Reports (DRs) and Record (DRs) and Record of Responses (RRs)of Responses (RRs)

• Amendments (AMs) Amendments (AMs) and Technical and Technical Corrigenda (TCs)Corrigenda (TCs)

DevelopmentConsensusBuilding

MAINTENANCEMAINTENANCE

5252


REVISE, REAFFIRM,REVISE, REAFFIRM, WITHDRAWWITHDRAW• Revise: new work Revise: new work

item, incorporate item, incorporate new technologynew technology

• Reaffirm: no Reaffirm: no changes, stable changes, stable technologytechnology

• Withdraw: little use, Withdraw: little use, obsolete technologyobsolete technology

• Typically: 5 year Typically: 5 year cyclecycle


MaintenanceREVISE, REVISE, REAFFIRM, REAFFIRM, WITHDRAWWITHDRAW

5353

IT Standards Life Cycle:IT Standards Life Cycle:Consistency via Accredited Consistency via Accredited

ProcessProcess• Accreditation affords Accreditation affords

consistent processconsistent process

• Committees don’t Committees don’t reinvent wheelreinvent wheel

• Choosing the right Choosing the right “process” is a critical “process” is a critical decisiondecision

• Accredited process is Accredited process is well-tested and well-tested and “off “off the shelf”the shelf”


MaintenanceRevise, Reaffirm, Withdraw

5454

IT Standards Life Cycle:IT Standards Life Cycle:Typical TimeframeTypical Timeframe

• Development: 12-48 mo.Development: 12-48 mo.

• Consensus: 9-24 mo.Consensus: 9-24 mo.

• Maintenance: 3-6 yrsMaintenance: 3-6 yrs

• Revise, reaffirm, Revise, reaffirm, withdraw: every 5 yearswithdraw: every 5 years

• Typical time, committee Typical time, committee formed to approved formed to approved standard: 18-48 monthsstandard: 18-48 months

• Realistic schedule, good Realistic schedule, good resultsresults


MaintenanceRevise, Reaffirm, Withdraw

5555

A SUGGESTIONA SUGGESTIONFOR A NEWFOR A NEW

INFORMATION INFORMATION TECHNOLOGYTECHNOLOGY

LIFE-CYCLELIFE-CYCLEMANAGEMENTMANAGEMENT

PARADIGMPARADIGM

5656

TestingTesting

Test Requirements

Abstract

Test Cases

Test System

and Tools

Test Prototypes

Test System

Validation

Dev. Test Services

Accreditation and

Certification

5757

ImplementationImplementation

Implementation Implementors Work Shop

(WS)

Product Planning

Prototypes Conformance TestingDem./Val. Dem./Val.

Tech Transfer of Test ToolsUser/Vendor Agreements

Training, Awareness, Market Building

Product Development

5858

The Standards Life-CycleThe Standards Life-CycleUser / Vendor

Needs AssessmentInternational Standard (IS)

Working Draft (WD)

Development

Committee Draft (CD)

Consensus and Verification

Draft International

Standard (DIS)

Validation, Consensus and FixesStandards

Implementation Implementors Work Shop (WS)

Product Planning

Prototypes Conformance Testing

Demonstration Validation



Product Development

Accreditation and

Certification

Test Requirements

Abstract Test Cases

Test System and Tools

Test Prototypes

Test System Validation

Dev. Test Services

Interoperability Testing

Registration Procurement

Testing

Integration

5959

Standards and Specifications Standards and Specifications CertificationCertification

User / Cust. Requirements

Technical Analysis

Cost / Benefit Analysis

Working Draft

Committee Draft

DIS

IS

ACCREDITATION & CERTIFICATION

CONSENSUS & CERTIFICATION

SPECIFICATION DEVELOPMENT

Test Requirements

Test Prototype

Validation Testing

Interoperability Testing

Conformance Testing

Develop/Validate

REVISIONS CYCLE

• ABSTRACT TEST CASES• TEST SYSTEMS/TOOLS

• PROTOTYPES

6060

Examples of Organizations Involved in the Examples of Organizations Involved in the Life-Cycle StagesLife-Cycle Stages

Stage 1:Initial

Requirements

Users

Vendors

Consortia

Etc.

Stage 2:Base

Standards Development

INCITS

SC4

OASIS

W3C

IETF

ISO/IEC JTC1

IEEE

EIA/TIA

Etc.

Stage 3:Profiles/ Product

Development

OIW

EWOS

AOW

MAP/TOP

Vendors

JTC1/SGFS

Consortia

Etc.

Stage 4:Testing

COS

NVLAP

EOTC

Vendors

Etc.

Stage 5:User

Implementation Feedback

END-USERS

Vendors

Etc.

RequirementsBase

Standards

ISPs

Tests

Changes

Standards/

Products

Test Results

6161

Examples of Organizations Involved in the Examples of Organizations Involved in the Life-Cycle StagesLife-Cycle Stages

Profiles (Interpretations, Changes, Enhance-ments)

Base Standards

Stage 3:Profiles/ Product

Development

Stage 2:Base

Standards Development

Stage 1:Initial

Requirements

Stage 5:User

Implementation Feedback

Stage 4:Testing

Base

Standards

ISPs

Tests

Changes

Standards/

Products

Test Results

Testing OrganizationsVendors

ConsortiaUsers

SDOs



ConsortiaUsers

SDOs

Requirements

New Requirements / New Technology

New Requirements

Addenda, New Standards

VendorsNew Technology

Test Results (Interpretations, Changes, Enhancements)

Feedback (Interpretations, Changes, Enhancements)

6262

Concurrent Test Concurrent Test Development Development • Requirements BasedRequirements Based• Concurrent Development of Test Suites & Concurrent Development of Test Suites &

Prototypes Prototypes – Early exposure of errors in initial specificationEarly exposure of errors in initial specification

• Early Error Correction in Prototype DesignEarly Error Correction in Prototype Design– Saves costs in later integration phaseSaves costs in later integration phase

• Rigorous Testing Reveals ErrorsRigorous Testing Reveals Errors– Specification AmbiguitiesSpecification Ambiguities– Specification OmissionsSpecification Omissions

• Early Interoperation of Tested SystemsEarly Interoperation of Tested Systems– Exposes errors in specification & in Test SuiteExposes errors in specification & in Test Suite

(Iterative Process)

6363

MetricsMetrics• Cost / ScheduleCost / Schedule

– Development / Implementation CostsDevelopment / Implementation Costs– Estimated Savings / BenefitsEstimated Savings / Benefits

• PerformancePerformance– Degree of Requirements SatisfactionDegree of Requirements Satisfaction

• UtilityUtility– Measure of Acceptance / SupportMeasure of Acceptance / Support

• StabilityStability– Key measure for a Standard / SpecificationKey measure for a Standard / Specification

• Vendors do not want to invest in an unstable Vendors do not want to invest in an unstable StandardStandard

• Users do not want to buy products that may become Users do not want to buy products that may become obsolete due to Standards Changeobsolete due to Standards Change

6464

Iterative Quality AssuranceIterative Quality Assurance

• Top-level management commitment / supportTop-level management commitment / support

• Each task subject to Quality AuditEach task subject to Quality Audit

• Continuous feedback / process improvementContinuous feedback / process improvement

• Pragmatic MetricsPragmatic Metrics

• Tracking MechanismTracking Mechanism– Tracking disposition of test faults, integration and Tracking disposition of test faults, integration and

demonstration problem reports, fixes, regression demonstration problem reports, fixes, regression testing of prototype subsystems & modulestesting of prototype subsystems & modules

6565

Incremental DevelopmentIncremental Development• Iterative process of specification, test Iterative process of specification, test

development, prototyping, & multi-vendor development, prototyping, & multi-vendor demonstrations to verify correctness and to demonstrations to verify correctness and to validate against requirementsvalidate against requirements

• Based on methods & techniques / procedures Based on methods & techniques / procedures proven successfulproven successful

• Modeling, analytical studies, technical reviews & Modeling, analytical studies, technical reviews & user/vendor workshopsuser/vendor workshops

• Continuous FeedbackContinuous Feedback

Requirements DevelopmentRequirements Development

Requirements Prototype DemonstrationsRequirements Prototype Demonstrations

6666

Standards / Specification Standards / Specification DevelopmentDevelopment

• Incremental DevelopmentIncremental Development

• Iterative Quality AssuranceIterative Quality Assurance

• Concurrent Test DevelopmentConcurrent Test Development

• Multi-vendor Prototype Multi-vendor Prototype

DemonstrationsDemonstrations

(Total Life-Cycle Process)

6767

Process CharacteristicsProcess Characteristics

• UnderstoodUnderstood

• VisibleVisible

• StructuredStructured

• MeasurableMeasurable

• AccountableAccountable

6868

The Key to The Key to

SuccessSuccess

is a is a

Well-Managed Well-Managed

Process!Process!

6969

Standards / Specification Standards / Specification CriteriaCriteria• Complete & UnambiguousComplete & Unambiguous• Achievable / Reliable / RealisticAchievable / Reliable / Realistic

– Can be developed / produced - there is a need for itCan be developed / produced - there is a need for it• VerifiedVerified

– Correct; internally consistentCorrect; internally consistent• ValidatedValidated

– Validated against user/customer requirements and Validated against user/customer requirements and needs in terms of:needs in terms of:• ReliabilityReliability• CommonalityCommonality• PerformancePerformance• OpennessOpenness• InteroperabilityInteroperability• CostCost

(Design / Development Goals)

7070

Total Life-Cycle ApproachTotal Life-Cycle Approach

• Unambiguous InterpretationUnambiguous Interpretation• Cost Effective SolutionCost Effective Solution• Truly Interoperable SystemsTruly Interoperable Systems• Proven & Stable Foundation for:Proven & Stable Foundation for:

– GovernmentGovernment– IndustryIndustry– AcademiaAcademia

• Quality Product that Vendors have Quality Product that Vendors have faith infaith in– Can depend on for marketing, Can depend on for marketing,

manufacturing, procurement, liabilitymanufacturing, procurement, liability– Produce conforming COTSProduce conforming COTS

Fundamental to Write a Specification which Results in:

7171

Multi-Vendor Prototype Multi-Vendor Prototype DemosDemos• Interactive ProcessInteractive Process• Direct feedback to development/refinement processDirect feedback to development/refinement process

– Refinement of working draftRefinement of working draft

• Use draft standard/specification to develop prototypeUse draft standard/specification to develop prototype• Use prototype for demonstration/validation exerciseUse prototype for demonstration/validation exercise• Test failures (attributable to):Test failures (attributable to):

– Prototype implementationPrototype implementation– Test systemsTest systems– Standard / SpecificationStandard / Specification

• Regression Tests performed when:Regression Tests performed when:– Standard/Specification is modifiedStandard/Specification is modified– Test systems are modifiedTest systems are modified

7272

Test ProcessTest ProcessStandard Develop

• Test Procedures• Test Systems

TEST

Test Error

Implementor Source of

Error

Log Problem Assign Priority

TRC Classify Error

C

B StandardError

Other Error

Prototype Demo• Integration• Interoperability• Interchangeability Testing

Failed Test Results Failed Test Results

IUTs (Modules,

Subsystems)

Tested IUTs

Fix IUTIUT Error

IUT Error

1 2

3

3

4

7373

Test Process Test Process (Cont’d)(Cont’d)

Log Problem• Agreement• Standard

B

Quick Fix

Test Case Fix

C

Generate Error

Report

STD. Development

Version Control• Errata

1

2

4

Test System Fix

Version Control• Errata

C

Test Expert: Classify

Error/Change

Develop Regression

Tests

Test Case Test System Software

3 3

Test Errors SOSAS Changes

7474

The IT Standards Life-CycleThe IT Standards Life-CycleUser /

Vendor

Needs Assessment

International

Standard (IS)

Working

Draft (WD)

Development

Committee

Draft (CD)

Consensus and Verification

Draft

International

Standard (DIS)

Validation, Consensus and FixesStandards

ImplementationImplementors

Work Shop (WS)

Product

Planning

PrototypesConformance

TestingDemonstration

Validation



Accreditation CertificationTest

Requirements Abstra

ct

Test Cases Test

Prototypes Test System

Validation

Dev. Test Services

Interoperability

Testing

Registration Procurement Integration

Testing

Product Development

1 4

2 3

5

6

7

Test System and Tools

7575

STANDARDSSTANDARDSTUTORIALTUTORIAL

Part VI – Some Part VI – Some ObservationsObservations

7676

Observations of Cultural DifferencesWith Respect to Standards Compliance

U.S.

Country Requirement Compliance Rules

Germany

Russia

France

Permitted

Prohibited

Prohibited

Permitted

EXCEPTEXCEPT

EXCEPTEXCEPT

EVEN

EVEN

Prohibited

Permitted

Permitted

PROHIBITED!

7777

Experience/ObservationsExperience/Observations

• GoodGood

• BadBad

• Lessons to be learnedLessons to be learned

7878

INDUSTRY EXPERIENCEINDUSTRY EXPERIENCE

• Success AttributesSuccess Attributes

• Failure Failure CharacteristicsCharacteristics

• ConclusionConclusion

7979

Success Attributes Success Attributes [1][1]

• Participants: expect long-term Participants: expect long-term involvement (years) involvement (years)

• Schedule: don’t get rushed - don’t get Schedule: don’t get rushed - don’t get late!late!

• New technology: be conservativeNew technology: be conservative

• Scope: obstinately stick to it!Scope: obstinately stick to it!

8080

Success Attributes Success Attributes [2][2]

• Conformance:Conformance:– need to measure itneed to measure it– should have working definition ASAPshould have working definition ASAP

• Target audience: commercial Target audience: commercial systems and userssystems and users

• Quality: fix bugs immediately!Quality: fix bugs immediately!• Process: have faith in consensus Process: have faith in consensus

process -- it works!process -- it works!

8181

Failure Attributes Failure Attributes [1][1]

• Incorporate new/untried technologyIncorporate new/untried technology– Why waste committee time?Why waste committee time?

• Ignore commercial interestsIgnore commercial interests– Who will implement the standard?Who will implement the standard?

• Ignore public commentsIgnore public comments– Who will buy standardized products?Who will buy standardized products?

• Creeping featurismCreeping featurism– The schedule killer!The schedule killer!

Failures: only recognized years Failures: only recognized years laterlater

8282

Failure Attributes [2]Failure Attributes [2]

• Poor time estimatesPoor time estimates– Progress is made over quarters, not weeksProgress is made over quarters, not weeks

• Leave bugs to laterLeave bugs to later– Expensive to fix later, like softwareExpensive to fix later, like software

• Weak tests of conformanceWeak tests of conformance– Standard-conforming, but lacks Standard-conforming, but lacks

interoperabilityinteroperability• Too much implementation-defined Too much implementation-defined

behaviorbehavior– dittoditto

Failures: only recognized years Failures: only recognized years laterlater

8383

Some ExamplesSome Examples• IEEE networking standards:IEEE networking standards:

– Success: widely acceptedSuccess: widely accepted• Supercomputing/parallelism features:Supercomputing/parallelism features:

– Failure: new technology, changing conceptual modelFailure: new technology, changing conceptual model– No problems standardizing small features, but big No problems standardizing small features, but big

picture wasn’t rightpicture wasn’t right– Hindsight: lack of big picture, technology hadn’t Hindsight: lack of big picture, technology hadn’t

matured, wasn’t asking right questionsmatured, wasn’t asking right questions• C++ standardization process:C++ standardization process:

– Failure: years late, creeping featurism, no scope Failure: years late, creeping featurism, no scope control, no rationale, no complete implementations control, no rationale, no complete implementations during development, many inconsistencies (while during development, many inconsistencies (while close to publishing standard)close to publishing standard)

– Expect many defect reports, poor conformance testsExpect many defect reports, poor conformance tests– Result: creation of JavaResult: creation of Java

8484

CONCLUSIONS CONCLUSIONS [1][1]

• Characteristics of Good technical work: Characteristics of Good technical work: – clear scopeclear scope– ““do-able”do-able”– support by vendorssupport by vendors– support by users support by users – well-defined conformance testswell-defined conformance tests

Basis: Basis: [experience][experience]

8585

CONCLUSIONS [2]CONCLUSIONS [2]

• Standards participation is long-term Standards participation is long-term commitment, but has high valuecommitment, but has high value

• Collaboration and liaising help reduce Collaboration and liaising help reduce duplicated effortsduplicated efforts

• Good technical standards take a Good technical standards take a while to “bake”while to “bake”

8686

CONCLUSIONS [3]CONCLUSIONS [3]

• Knowledge of the standards process can Knowledge of the standards process can be very helpful for internal projects:be very helpful for internal projects:– Specification development and consensus-Specification development and consensus-

building techniques are widely usefulbuilding techniques are widely useful– Quality is recognized at the end with few Quality is recognized at the end with few

defect reports and consistent spec defect reports and consistent spec interpretationinterpretation

– Standards process is a “best practice” to Standards process is a “best practice” to develop high quality specs within a develop high quality specs within a reasonable technical horizonreasonable technical horizon

8787

OUR EXPERIENCEOUR EXPERIENCE

• De jure vs Consortia Process:De jure vs Consortia Process:

• Organizations don’t actually compete - Organizations don’t actually compete - each has a role, scope, and purposeeach has a role, scope, and purpose– Consortia best rapid for technology Consortia best rapid for technology

developmentdevelopment– Formal de jure process best for consensus-Formal de jure process best for consensus-

buildingbuilding– but not vice versabut not vice versa– Best of Both Worlds ExamplesBest of Both Worlds Examples

IT STANDARDS TUTORIAL Part IV – Standardization Processes

Documents

Transcript of IT STANDARDS TUTORIAL Part IV – Standardization Processes