STANDARDS, TESTING AND VALIDATION · 2017-12-06 · Achieving Interoperability – Best Practices...
Transcript of STANDARDS, TESTING AND VALIDATION · 2017-12-06 · Achieving Interoperability – Best Practices...
Partners:
STANDARDS, TESTING AND VALIDATIONPresented by Sebastian MüllerProject Manager ETSI Centre for Testing and Interoperability © All rights reserved
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
• ETSI – A member‐driven Standards Organization (SDO) setting globally‐applicable standards in ICT since 1988
• Whether it is from:• LTE to Multi‐access Edge Computing; or• Digital Broadcasting to Network Functions Virtualisation; or• Emergency Telecommunications to the Internet of Things …
• … ETSI strives to produce high‐quality, relevant, timely, testable and interoperable standards
• Our members have the same ambitions for their products based on ETSI standards
• This presentation is about how together we achieve these goals• But first, a few words about what we do …
ABOUT THIS PRESENTATION
2
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
• ETSI – A member‐driven Standards Organization (SDO) setting globally‐applicable standards in ICT since 1988
GLOBAL STANDARDS COLLABORATION
(Canada)
(USA)
(USA)
(International)
(Japan)
(Korea)
(Japan)
(China)
(USA)(India)
3
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
VARIOUS TECHNICAL GROUPS26 Technical Committees (TC)• ATTM, BRAN, BROADCAST, CABLE, CYBER, DECT, EE, ERM, ESI, HF,
INT, ITS, LI, MSG, MTS, NTECH, PLT, RRS, RT, SAFETY, SCP, SES, SmartBAN, SmartM2M, STQ, TCCE.
4 Special groups• EP eHealth, SC EMTEL, SC SAGE, SC USER. 2 ETSI Partnership projects• 3GPP, oneM2M.14 Industry Specification Groups (ISG)• CCM, CIM, ECI, ENI, IP6, ISI, MBC, MEC, mWT, NFV, NGP, OEU,
QKD, SMT.1 Open Source Group (OSG)• OSM
4
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
ETSI DELIVERABLES AND APPROVALEN – European Standard: Normative• Approval level: NSO (minimum 90 days)EN – Harmonized Standard: Normative• Approval level: NSO + European Commission review (minimum 120 + 60 days)ES – ETSI Standard: Normative• Approval level: ETSI Membership (90 days)EG – ETSI Guide: Informative (guidance)• Approval level: ETSI Membership (Immediate)TS – ETSI Technical Specification: Normative• Approval level: relevant Technical Committee (Immediate)TR – ETSI Technical Report: Informative (informational)• Approval level: relevant Technical Committee (Immediate)GS – ETSI Technical Specification: Normative• Approval level: relevant Industry Specification Group (Immediate)GR – ETSI technical Report: Informative• Approval level: relevant Industry Specification Group (Immediate)
Harmonized Standards are ENs with a special status. They are produced in response to an EC mandate. They provide the technical detail necessary to achieve the ‘essential requirements’ of an EC Directive. Harmonised betweed CEN, CENELEC and ETSI. They are thus key enablers of the European Single Market.
5
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
STANDARDS ‘ECOSYSTEM’ (much SIMPLIFIED!)
6
ITU
ETSI
… to deliver standards that enable ...• Testable• Interoperable• Reliable• High‐quality ...... products
IEEE
IETF
Fora
Working with Vendors …
Working with Operators, Service Providers and Users …
Partnerships3GPP
OneM2M
Working with Partners …
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
OUR ROLE IN THE STANDARDS MAKING PROCESS
Validation and
ReviewDrafting Editorial
Check
Approval and
Publication
Work Item
Feedback
Feedback
Feedback
Published
Standard
Implementationand use
Maintenance and evolution resulting in a new Work Item for a revision of the standard
CTI is active here ... ... to influence here
7
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
OUR APPROACH TO ENSURE HIGH‐QUALITY, INTEROPERABLE STANDARDS
Implementations mature from prototypes to INTEROPERABLE commercial products
Market Awareness and
Pre‐Standardization:
Proofs of Concepts
Testing and Validation
Methodologies test techniques, frameworks,
repositories and tools
INTEROPERABLE standards evolve in parallel with product development
Standardized Test SpecificationsConformance, Interoperability etc.
PLUGTESTS EventsInteroperability, pre‐conformance, coexistence etc.
Product Testing (not done by ETSI)
Certification (not done by ETSI)
8
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
• Trial implementations built around well‐defined scenarios• demonstrate viability of standards solutions• provide feedback to guide the development of the standards• help build awareness and confidence in the technology• help develop diverse and open ecosystems
• Usually small groups (2‐3 vendors + 1‐2 operators)• exhibits at specific events• demonstrations in laboratories• experiences from (restricted) real deployment situations
• Successfully used by• ISG NFV (over 40 PoCs running) http://www.etsi.org/technologies‐clusters/technologies/nfv/nfv‐poc
• ISG MEC (4 PoCs running ‐more on the way)• TC NTECH/AFI (PoCs programme just initiated)
PROOFS OF CONCEPT (PoC)
9
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
INTEROPERABILITY TESTING
Tests end‐to‐end functionality between equipment or systems
11
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
WHY ETSI INTEROPERABILITY EVENTS?
11
• Foster interoperability
• Gather feedback• Validate and improve standards
• Debug and validate implementations (products)
• Free and open to anyone who has an implementation to test
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
• Over 200 events since 1999• 10 ‐ 15 events per year• Events on‐site and/or remote• Over 60 different technologies• http://www.etsi.org/about/what‐we‐do/plugtests
ETSI INTEROPERABILITY EVENTS LOOK LIKE THIS
12
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
• Hackathons• Recent initiative (for ETSI)• Combining Plugtests events with hackathon activity• Usually dedicated to addressing a specific technical ‘problem(s)’• Quick resolution of bugs found in the standard or implementations during the test event
• Specfests/Hackfests• Mostly oriented to APIs• Developer guides• Hands‐on tutorials• Bringing developers and standardizers together
HACKATHONS AND SPECFESTS/HACKFESTS
15
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
CONFORMANCE TESTING
Tests equipment for compliance to requirements in a base standard
16
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
DEVELOPMENT OF CONFORMANCE TEST SPECS
WHAT to test HOW to test(high‐level)
Test‐relevant behaviourand data
HOW to test(detailed)
Complete behaviourComplete data
Verdict assignment
Pre‐Conditions
Cleanup
Test Purposes
Compiled and Validated Executable Code
(binary)
Full protocol and system knowledge
Run‐time support
Test Descriptions Test Cases Executable Test Cases
Prose or TPLAN
Proseor TDLExtracted
Requirements
TTCN‐3(Source)Base Standard(s)
Certification
17
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
ALIGNING WITH THE STANDARDS DEVELOPMENT PROCESS
R0 R1
Validation Testing
Test Descriptions
Test Purposes
Plugtest#1
Test Cases
R2
R1 Products
R1Test Spec
Plugtest#2 Plugtest#3
19
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
SUMMARY (OR WHY WE ARE DOING THIS)
PoCsAwareness (external)Feasibility, relevance, timeliness of standardsProposed standardization topics… ...interoperable
solutions
PlugtestsCommunity building
(standards makers and implementors together)Validated standards
Tested (debugged) productsFeedback to testers and standardizers
Interoperable products……more likely to conform
Conformance TestingInternal proprietary testing
RegulationCertification
Conformant products……more likely to interoperate
Hackathons and SpecfestsInvestigate specific (technical) issues
EducationTaking the standards to
developers
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
• Air Traffic Management VoIP (with EUROCAE) events• Intelligent Transport Systems (ITS) events and conformance test specifications and frameworks
• Next Generation Emergency Communications (NG112) events• eCall events (with partner ERTICO)• Mission Critical Push‐to‐Talk (MCPTT) event (with partner TCCA)• oneM2M events (with partner TTA), conformance test specifications, user guides and hackathons
• Network Functions Virtualization (NFV)• Interop, API testing, Open Source project ...
• 3GPP UE certification tests
SOME ‘SUCCESS STORIES’
18
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
• Capitalising on synergy between Standardizers – Vendors –Users to deliver standards that are• Interoperable, timely, testable, high‐quality, relevant etc.
• Pre‐standardization activities are becoming an integral part of the standardization process
• Validated standards are key to interoperable products• Testing and certification is crucial in key technologies
• 3GPP LTE, ITS, oneM2M, DECT, TETRA ...• Interop, conformance and indirectly performance, robustness, reliability ...
• Need to encompass Open Source activities and approaches...
TAKEAWAYS
20
ETSI Symposium onAchieving Interoperability – Best Practices in Standardization
Sebastian Müller on behalf of
Anthony WilesDirector
Centre for Testing and Interoperability European Telecommunications Standards Institute
Thank you!