Promoting and Standardizing Grid Computing
Grid and Distributed Software Certification and Quality AssuranceGGF17 Certification BoFMarc-Elian BéginCERN
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 2
Agenda
• Introduction
• ETICS overview
• OMII (Steven Newhouse)
• Discussion
• Conclusion
www.eu-etics.org
INFSOM-RI-026753
ETICS: E-infrastructure for ETICS: E-infrastructure for Testing, Integration and Testing, Integration and Configuration of SoftwareConfiguration of Software
Marc-Elian BéginMarc-Elian BéginDissemination, certification and web tools Work-Dissemination, certification and web tools Work-package Leaderpackage LeaderCERNCERN
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 4INFSOM-RI-026753
Vision and Mission
• VisionVision: A dependable, reliable, stable grid : A dependable, reliable, stable grid infrastructure requires high-quality, thoroughly infrastructure requires high-quality, thoroughly tested, interoperable software middleware and tested, interoperable software middleware and applicationsapplications
• MissionMission: Provide a generic service that other : Provide a generic service that other projects can use to efficiently and easily build projects can use to efficiently and easily build and test their grid and distributed software.and test their grid and distributed software.Set up the foundations for a certification Set up the foundations for a certification process to help increasing the quality and process to help increasing the quality and interoperability of such softwareinteroperability of such software
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 5INFSOM-RI-026753
Short Term Use Cases
• Automatically build software on different platformsAutomatically build software on different platforms• Automatically collect and store metrics during build Automatically collect and store metrics during build
processprocess• Quality Assurance trend analysis on collected metricsQuality Assurance trend analysis on collected metrics• Automatic handling and management of dependenciesAutomatic handling and management of dependencies• Automatic execution of testsuites, including build, Automatic execution of testsuites, including build,
installation and configuration of dependenciesinstallation and configuration of dependencies• ““Shop” dependencies from quality measured artifacts Shop” dependencies from quality measured artifacts
(components, services and plug-in) via managed (components, services and plug-in) via managed repositoryrepository
• Offload deployment of complex middleware servicesOffload deployment of complex middleware services• Combinations of the aboveCombinations of the above
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 6INFSOM-RI-026753
ETICS in a Nutshell
• ETICS stands forETICS stands foree-Infrastructure for -Infrastructure for TTesting, esting, IIntegration and ntegration and CConfiguration of onfiguration of SSoftwareoftware
• It’s an SSAIt’s an SSA• It has been granted a contribution of 1.4 MIt has been granted a contribution of 1.4 M€€• It has a duration of two yearsIt has a duration of two years• The project has started on January 1The project has started on January 1stst, 2006, 2006
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 7INFSOM-RI-026753
The ETICS Partners
Build system,software configuration, service infrastructure, dissemination, EGEE,
gLite, project coord.
Software configuration, service infrastructure, dissemination
Web portals and tools, quality process, dissemination, DILIGENT
Test methods and metrics, unit testing tools, EBIT
The Condor batch system, distributed testing tools, service infrastructure, NMI
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 8INFSOM-RI-026753
ETICS Objectives
• Objective 1 (technical)Objective 1 (technical)– Provide a comprehensive build and test service especially Provide a comprehensive build and test service especially
designed for grid softwaredesigned for grid software– Support multi-platform, distributed operations to build software Support multi-platform, distributed operations to build software
and run complex test cases (functional, regression, performance, and run complex test cases (functional, regression, performance, stress, benchmarks, interoperability, etc)stress, benchmarks, interoperability, etc)
• Objective 2 (coordination, policies)Objective 2 (coordination, policies)– Establish the foundations for a certification processEstablish the foundations for a certification process– Contribute to interoperability of grid middleware and applications Contribute to interoperability of grid middleware and applications
by promoting consistent build and test procedures and by by promoting consistent build and test procedures and by easying the verification of compliance to standardseasying the verification of compliance to standards
– Promote sound QA principles adapted to grid environment Promote sound QA principles adapted to grid environment through the participation to conferences, workshops, computing through the participation to conferences, workshops, computing training events (CSC, ICEAGE, EGEE)training events (CSC, ICEAGE, EGEE)
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 9INFSOM-RI-026753
Service Overview
Build/TestArtefacts
Web Application
ReportDB
ProjectDB
NMI Scheduler
Clients
Web Service
NMI ClientWrapper
Via browser
Via command-Line tools
WNs ETICS Infrastructure
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 10INFSOM-RI-026753
User roles
• GuestGuest: can execute local builds and test of modules: can execute local builds and test of modules• DevelopersDevelopers: can create configurations and execute : can create configurations and execute
local and remote build and test of moduleslocal and remote build and test of modules• IntegratorsIntegrators: can register build artefacts in repository: can register build artefacts in repository• TestersTesters: can register and perform trend analysis of test : can register and perform trend analysis of test
resultsresults• Release ManagersRelease Managers: can publish and release artefacts, : can publish and release artefacts,
create release notes and other documentationcreate release notes and other documentation• Module AdministratorModule Administrator: create modules and assign : create modules and assign
permissions with a given projectpermissions with a given project• AdministrationAdministration: can create and manage projects, : can create and manage projects,
assign permissions, set policies, etcassign permissions, set policies, etc
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 11INFSOM-RI-026753
QA Certification Process
• It’s the other major goal of the projectIt’s the other major goal of the project• ETICS must deliver a ETICS must deliver a feasibility studyfeasibility study for the for the
implementation of a QA certification process for grid implementation of a QA certification process for grid and distributed softwareand distributed software
• Not just a document, but a Not just a document, but a full assessmentfull assessment of what of what procedures, tools and rules maximize the chances of procedures, tools and rules maximize the chances of project of delivering reliable, interoperable software project of delivering reliable, interoperable software based on the real data collected in two years of activitybased on the real data collected in two years of activity
• We will propose it as the starting point of a coordinated We will propose it as the starting point of a coordinated QA Certification activityQA Certification activity in the context of a more in the context of a more permanent grid infrastructure initiativepermanent grid infrastructure initiative
• Will be done in close collaboration with the ETICS Will be done in close collaboration with the ETICS Experts Board to make sure it is practical and realisticExperts Board to make sure it is practical and realistic
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 12INFSOM-RI-026753
Standardisation effort
• e-Infrastructure Reflection Group (e-IRG)e-Infrastructure Reflection Group (e-IRG)– e-IRG mandate: “coordinates on a high European level the e-IRG mandate: “coordinates on a high European level the
introduction of a (grid based) infrastructure for e-Science”introduction of a (grid based) infrastructure for e-Science”– Working on raising awareness inWorking on raising awareness in
– Software Quality Assurance and automationSoftware Quality Assurance and automation– Standard software Certification ProcessStandard software Certification Process
– Co-chaired a working session during last e-IRG meeting in Linz Co-chaired a working session during last e-IRG meeting in Linz on “Middleware Interoperability and Repository”on “Middleware Interoperability and Repository”
• Global Grid Forum (GGF)Global Grid Forum (GGF)– GGF mandate: “international community leading the pervasive GGF mandate: “international community leading the pervasive
adoption of grid computing in research and industry”adoption of grid computing in research and industry”– Chairing a working session in the next GGF meeting in Tokyo on Chairing a working session in the next GGF meeting in Tokyo on
““Grid and Distributed Software Certification and Quality Grid and Distributed Software Certification and Quality AssuranceAssurance””
– Raise awareness on the need for Standard Certification Raise awareness on the need for Standard Certification Processes, including automated tests for interoperabilityProcesses, including automated tests for interoperability
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 13INFSOM-RI-026753
Typical ETICS working scenario
OMIIEU/UK
HEP \ BIO \INDUSTRY
ETICS OMIIEU/UK
Repositories
Other(projectspecific)
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 14INFSOM-RI-026753
Repositories
• All artifacts produced when using the ETICS services All artifacts produced when using the ETICS services are made available to the community at largeare made available to the community at large– Source and binary packagesSource and binary packages– Test reportsTest reports– Quality assurance reportsQuality assurance reports– BenchmarksBenchmarks– Interoperability evaluations, etcInteroperability evaluations, etc
• ETICS is involved in digital libraries and repositories ETICS is involved in digital libraries and repositories on two fronts:on two fronts:– ETICS has among its partners DILIGENT, a Grid powered Digital ETICS has among its partners DILIGENT, a Grid powered Digital
LibraryLibrary– ETICS is collaborating with the EU-OMII project to set up a ETICS is collaborating with the EU-OMII project to set up a
reference repository of quality assured software packagesreference repository of quality assured software packages
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 15INFSOM-RI-026753
Virtual Test Lab
• The Grid must embrace diversity in terms of platforms, The Grid must embrace diversity in terms of platforms, development languages, deployment methods, etcdevelopment languages, deployment methods, etc
• However, testing all resulting combinations is very However, testing all resulting combinations is very difficult and time consuming, not to mention the difficult and time consuming, not to mention the manpower requiredmanpower required
• ETICS is proposing a high-efficiency automation ETICS is proposing a high-efficiency automation process, but still providing and especially maintaining process, but still providing and especially maintaining the required hardware and software resources is not the required hardware and software resources is not trivialtrivial
• A possible solution is the use of virtualisationA possible solution is the use of virtualisation
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 16INFSOM-RI-026753
Virtual Test Lab
ETICS Engine
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 17INFSOM-RI-026753
The Virtual Test Lab
ETICS EngineMicrosoft
Virtual Server
Images can contain preinstalled OSs in fixed, reproducible configurations
Security risks are minimized, the image is destroyed when the test is over. But it can also be archived for later offline analysis of the tests
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 18INFSOM-RI-026753
Example of Web Application
Project structureProject structure
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 19INFSOM-RI-026753
Example of Web Application
Project configurationProject configuration
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 20INFSOM-RI-026753
Example of Build Report
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 21INFSOM-RI-026753
Example of Test Report
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 22INFSOM-RI-026753
Example of Job Submission Report
Preliminary integration of the web application with NMIPreliminary integration of the web application with NMI
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 23INFSOM-RI-026753
Example of Client
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 24INFSOM-RI-026753
Status – ETICS Development
• ClientClient– Available for local and remote building and testing software (read Available for local and remote building and testing software (read
only)only)– Editing feature under developmentEditing feature under development
• ServerServer– Available for building and testing software (read only)Available for building and testing software (read only)– Editing feature under developmentEditing feature under development– Secure access under developmentSecure access under development– Reports engine under developmentReports engine under development
• Web ApplicationWeb Application– Available for browsing ETICS metadataAvailable for browsing ETICS metadata– Editing feature under developmentEditing feature under development
• Editing stopgap solutionEditing stopgap solution– Tool for registering new projectsTool for registering new projects
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 25INFSOM-RI-026753
Status – Usage
• Build systemBuild system– Large element of gLite 3.0 buildingLarge element of gLite 3.0 building– On track to build entire gLite 3.1 stackOn track to build entire gLite 3.1 stack– In the process of building CERN/FIO Quattor softwareIn the process of building CERN/FIO Quattor software– In the process of building DILIGENT middleware/softwareIn the process of building DILIGENT middleware/software– Building ETICS softwareBuilding ETICS software
• Test systemTest system– Porting gLite Certification TestsuitesPorting gLite Certification Testsuites– Executing ETICS stress testsuiteExecuting ETICS stress testsuite
• DeploymentDeployment– UoW: ~70 machinesUoW: ~70 machines– CERN: ~20 machinesCERN: ~20 machines– INFN: ~3 machinesINFN: ~3 machines
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 26INFSOM-RI-026753
Potential new users
• Automatic execution of build/testAutomatic execution of build/test– MPI testsuites execution and trend analysis: requested by Cal MPI testsuites execution and trend analysis: requested by Cal
Loomis (EGEE/NA4)Loomis (EGEE/NA4)– Build and test of OMII components (UK+Europe)Build and test of OMII components (UK+Europe)
• CERN CERN Administrative Information ServicesAdministrative Information Services– Build and test of J2EE applicationsBuild and test of J2EE applications
• OthersOthers
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 27INFSOM-RI-026753
Long Term Future and Sustainability
• We envision ETICS to become a permanent service We envision ETICS to become a permanent service after its initial two-year phaseafter its initial two-year phase
• As projects start using and relying on it for managing As projects start using and relying on it for managing the software development cycle, the ETICS the software development cycle, the ETICS infrastructure should get enough “critical mass” to be infrastructure should get enough “critical mass” to be supported by research and industrial organisation as supported by research and industrial organisation as other “commodity” servicesother “commodity” services
• In addition, we want to propose ETICS as one of the In addition, we want to propose ETICS as one of the cornerstones of a more permanent international cornerstones of a more permanent international collaboration to establish a European and world-wide collaboration to establish a European and world-wide grid infrastructuregrid infrastructure
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 28INFSOM-RI-026753
Thanks
http://www.eu-etics.org
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 29
OMII
Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 30
Questions/Issues for discussion• How do you see certification and QA in your context?
− Do you need it?− Are you doing it?− If you don’t, why?− Would a “standardised certification process” be useful?
• Building− Do you do multi-platform build and test?− If so, how?− If not, who does it for you, or why not?− Where do you put your build artefacts?− Where do you find your dependencies and how to you manage them?− How do you manage your releases?
• Testing− Do you have testsuites? Are they automated?− Are you using tools to implement and execute your tests?− How do you execute them and how do you handle test outputs?− How do you analyse the results
• Scope for standardisation− Process− Test execution and output formats− Metrics and their collection
Top Related