Grid and Distributed Software Certification and Quality Assurance

Promoting and Standardizing Grid Computing

Grid and Distributed Software Certification and Quality AssuranceGGF17 Certification BoFMarc-Elian BéginCERN

Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 2

Agenda

• Introduction

• ETICS overview

• OMII (Steven Newhouse)

• Discussion

• Conclusion

www.eu-etics.org

INFSOM-RI-026753

ETICS: E-infrastructure for ETICS: E-infrastructure for Testing, Integration and Testing, Integration and Configuration of SoftwareConfiguration of Software

Marc-Elian BéginMarc-Elian BéginDissemination, certification and web tools Work-Dissemination, certification and web tools Work-package Leaderpackage LeaderCERNCERN

Grid and Distributed Software Certification and Quality Assurance - GGF17 - Tokyo 4INFSOM-RI-026753

Vision and Mission

• VisionVision: A dependable, reliable, stable grid : A dependable, reliable, stable grid infrastructure requires high-quality, thoroughly infrastructure requires high-quality, thoroughly tested, interoperable software middleware and tested, interoperable software middleware and applicationsapplications

• MissionMission: Provide a generic service that other : Provide a generic service that other projects can use to efficiently and easily build projects can use to efficiently and easily build and test their grid and distributed software.and test their grid and distributed software.Set up the foundations for a certification Set up the foundations for a certification process to help increasing the quality and process to help increasing the quality and interoperability of such softwareinteroperability of such software


Short Term Use Cases

• Automatically build software on different platformsAutomatically build software on different platforms• Automatically collect and store metrics during build Automatically collect and store metrics during build

processprocess• Quality Assurance trend analysis on collected metricsQuality Assurance trend analysis on collected metrics• Automatic handling and management of dependenciesAutomatic handling and management of dependencies• Automatic execution of testsuites, including build, Automatic execution of testsuites, including build,

installation and configuration of dependenciesinstallation and configuration of dependencies• ““Shop” dependencies from quality measured artifacts Shop” dependencies from quality measured artifacts

(components, services and plug-in) via managed (components, services and plug-in) via managed repositoryrepository

• Offload deployment of complex middleware servicesOffload deployment of complex middleware services• Combinations of the aboveCombinations of the above


ETICS in a Nutshell

• ETICS stands forETICS stands foree-Infrastructure for -Infrastructure for TTesting, esting, IIntegration and ntegration and CConfiguration of onfiguration of SSoftwareoftware

• It’s an SSAIt’s an SSA• It has been granted a contribution of 1.4 MIt has been granted a contribution of 1.4 M€€• It has a duration of two yearsIt has a duration of two years• The project has started on January 1The project has started on January 1stst, 2006, 2006


The ETICS Partners

Build system,software configuration, service infrastructure, dissemination, EGEE,

gLite, project coord.

Software configuration, service infrastructure, dissemination

Web portals and tools, quality process, dissemination, DILIGENT

Test methods and metrics, unit testing tools, EBIT

The Condor batch system, distributed testing tools, service infrastructure, NMI


ETICS Objectives

• Objective 1 (technical)Objective 1 (technical)– Provide a comprehensive build and test service especially Provide a comprehensive build and test service especially

designed for grid softwaredesigned for grid software– Support multi-platform, distributed operations to build software Support multi-platform, distributed operations to build software

and run complex test cases (functional, regression, performance, and run complex test cases (functional, regression, performance, stress, benchmarks, interoperability, etc)stress, benchmarks, interoperability, etc)

• Objective 2 (coordination, policies)Objective 2 (coordination, policies)– Establish the foundations for a certification processEstablish the foundations for a certification process– Contribute to interoperability of grid middleware and applications Contribute to interoperability of grid middleware and applications

by promoting consistent build and test procedures and by by promoting consistent build and test procedures and by easying the verification of compliance to standardseasying the verification of compliance to standards

– Promote sound QA principles adapted to grid environment Promote sound QA principles adapted to grid environment through the participation to conferences, workshops, computing through the participation to conferences, workshops, computing training events (CSC, ICEAGE, EGEE)training events (CSC, ICEAGE, EGEE)


Service Overview

Build/TestArtefacts

Web Application

ReportDB

ProjectDB

NMI Scheduler

Clients

Web Service

NMI ClientWrapper

Via browser

Via command-Line tools

WNs ETICS Infrastructure


User roles

• GuestGuest: can execute local builds and test of modules: can execute local builds and test of modules• DevelopersDevelopers: can create configurations and execute : can create configurations and execute

local and remote build and test of moduleslocal and remote build and test of modules• IntegratorsIntegrators: can register build artefacts in repository: can register build artefacts in repository• TestersTesters: can register and perform trend analysis of test : can register and perform trend analysis of test

resultsresults• Release ManagersRelease Managers: can publish and release artefacts, : can publish and release artefacts,

create release notes and other documentationcreate release notes and other documentation• Module AdministratorModule Administrator: create modules and assign : create modules and assign

permissions with a given projectpermissions with a given project• AdministrationAdministration: can create and manage projects, : can create and manage projects,

assign permissions, set policies, etcassign permissions, set policies, etc


QA Certification Process

• It’s the other major goal of the projectIt’s the other major goal of the project• ETICS must deliver a ETICS must deliver a feasibility studyfeasibility study for the for the

implementation of a QA certification process for grid implementation of a QA certification process for grid and distributed softwareand distributed software

• Not just a document, but a Not just a document, but a full assessmentfull assessment of what of what procedures, tools and rules maximize the chances of procedures, tools and rules maximize the chances of project of delivering reliable, interoperable software project of delivering reliable, interoperable software based on the real data collected in two years of activitybased on the real data collected in two years of activity

• We will propose it as the starting point of a coordinated We will propose it as the starting point of a coordinated QA Certification activityQA Certification activity in the context of a more in the context of a more permanent grid infrastructure initiativepermanent grid infrastructure initiative

• Will be done in close collaboration with the ETICS Will be done in close collaboration with the ETICS Experts Board to make sure it is practical and realisticExperts Board to make sure it is practical and realistic


Standardisation effort

• e-Infrastructure Reflection Group (e-IRG)e-Infrastructure Reflection Group (e-IRG)– e-IRG mandate: “coordinates on a high European level the e-IRG mandate: “coordinates on a high European level the

introduction of a (grid based) infrastructure for e-Science”introduction of a (grid based) infrastructure for e-Science”– Working on raising awareness inWorking on raising awareness in

– Software Quality Assurance and automationSoftware Quality Assurance and automation– Standard software Certification ProcessStandard software Certification Process

– Co-chaired a working session during last e-IRG meeting in Linz Co-chaired a working session during last e-IRG meeting in Linz on “Middleware Interoperability and Repository”on “Middleware Interoperability and Repository”

• Global Grid Forum (GGF)Global Grid Forum (GGF)– GGF mandate: “international community leading the pervasive GGF mandate: “international community leading the pervasive

adoption of grid computing in research and industry”adoption of grid computing in research and industry”– Chairing a working session in the next GGF meeting in Tokyo on Chairing a working session in the next GGF meeting in Tokyo on

““Grid and Distributed Software Certification and Quality Grid and Distributed Software Certification and Quality AssuranceAssurance””

– Raise awareness on the need for Standard Certification Raise awareness on the need for Standard Certification Processes, including automated tests for interoperabilityProcesses, including automated tests for interoperability


Typical ETICS working scenario

OMIIEU/UK

HEP \ BIO \INDUSTRY

ETICS OMIIEU/UK

Repositories

Other(projectspecific)


Repositories

• All artifacts produced when using the ETICS services All artifacts produced when using the ETICS services are made available to the community at largeare made available to the community at large– Source and binary packagesSource and binary packages– Test reportsTest reports– Quality assurance reportsQuality assurance reports– BenchmarksBenchmarks– Interoperability evaluations, etcInteroperability evaluations, etc

• ETICS is involved in digital libraries and repositories ETICS is involved in digital libraries and repositories on two fronts:on two fronts:– ETICS has among its partners DILIGENT, a Grid powered Digital ETICS has among its partners DILIGENT, a Grid powered Digital

LibraryLibrary– ETICS is collaborating with the EU-OMII project to set up a ETICS is collaborating with the EU-OMII project to set up a

reference repository of quality assured software packagesreference repository of quality assured software packages


Virtual Test Lab

• The Grid must embrace diversity in terms of platforms, The Grid must embrace diversity in terms of platforms, development languages, deployment methods, etcdevelopment languages, deployment methods, etc

• However, testing all resulting combinations is very However, testing all resulting combinations is very difficult and time consuming, not to mention the difficult and time consuming, not to mention the manpower requiredmanpower required

• ETICS is proposing a high-efficiency automation ETICS is proposing a high-efficiency automation process, but still providing and especially maintaining process, but still providing and especially maintaining the required hardware and software resources is not the required hardware and software resources is not trivialtrivial

• A possible solution is the use of virtualisationA possible solution is the use of virtualisation


Virtual Test Lab

ETICS Engine


The Virtual Test Lab

ETICS EngineMicrosoft

Virtual Server

Images can contain preinstalled OSs in fixed, reproducible configurations

Security risks are minimized, the image is destroyed when the test is over. But it can also be archived for later offline analysis of the tests


Example of Web Application

Project structureProject structure


Example of Web Application

Project configurationProject configuration


Example of Build Report


Example of Test Report


Example of Job Submission Report

Preliminary integration of the web application with NMIPreliminary integration of the web application with NMI


Example of Client


Status – ETICS Development

• ClientClient– Available for local and remote building and testing software (read Available for local and remote building and testing software (read

only)only)– Editing feature under developmentEditing feature under development

• ServerServer– Available for building and testing software (read only)Available for building and testing software (read only)– Editing feature under developmentEditing feature under development– Secure access under developmentSecure access under development– Reports engine under developmentReports engine under development

• Web ApplicationWeb Application– Available for browsing ETICS metadataAvailable for browsing ETICS metadata– Editing feature under developmentEditing feature under development

• Editing stopgap solutionEditing stopgap solution– Tool for registering new projectsTool for registering new projects


Status – Usage

• Build systemBuild system– Large element of gLite 3.0 buildingLarge element of gLite 3.0 building– On track to build entire gLite 3.1 stackOn track to build entire gLite 3.1 stack– In the process of building CERN/FIO Quattor softwareIn the process of building CERN/FIO Quattor software– In the process of building DILIGENT middleware/softwareIn the process of building DILIGENT middleware/software– Building ETICS softwareBuilding ETICS software

• Test systemTest system– Porting gLite Certification TestsuitesPorting gLite Certification Testsuites– Executing ETICS stress testsuiteExecuting ETICS stress testsuite

• DeploymentDeployment– UoW: ~70 machinesUoW: ~70 machines– CERN: ~20 machinesCERN: ~20 machines– INFN: ~3 machinesINFN: ~3 machines


Potential new users

• Automatic execution of build/testAutomatic execution of build/test– MPI testsuites execution and trend analysis: requested by Cal MPI testsuites execution and trend analysis: requested by Cal

Loomis (EGEE/NA4)Loomis (EGEE/NA4)– Build and test of OMII components (UK+Europe)Build and test of OMII components (UK+Europe)

• CERN CERN Administrative Information ServicesAdministrative Information Services– Build and test of J2EE applicationsBuild and test of J2EE applications

• OthersOthers


Long Term Future and Sustainability

• We envision ETICS to become a permanent service We envision ETICS to become a permanent service after its initial two-year phaseafter its initial two-year phase

• As projects start using and relying on it for managing As projects start using and relying on it for managing the software development cycle, the ETICS the software development cycle, the ETICS infrastructure should get enough “critical mass” to be infrastructure should get enough “critical mass” to be supported by research and industrial organisation as supported by research and industrial organisation as other “commodity” servicesother “commodity” services

• In addition, we want to propose ETICS as one of the In addition, we want to propose ETICS as one of the cornerstones of a more permanent international cornerstones of a more permanent international collaboration to establish a European and world-wide collaboration to establish a European and world-wide grid infrastructuregrid infrastructure


Thanks

http://www.eu-etics.org


OMII


Questions/Issues for discussion• How do you see certification and QA in your context?

− Do you need it?− Are you doing it?− If you don’t, why?− Would a “standardised certification process” be useful?

• Building− Do you do multi-platform build and test?− If so, how?− If not, who does it for you, or why not?− Where do you put your build artefacts?− Where do you find your dependencies and how to you manage them?− How do you manage your releases?

• Testing− Do you have testsuites? Are they automated?− Are you using tools to implement and execute your tests?− How do you execute them and how do you handle test outputs?− How do you analyse the results

• Scope for standardisation− Process− Test execution and output formats− Metrics and their collection

Grid and Distributed Software Certification and Quality Assurance

Documents

Transcript of Grid and Distributed Software Certification and Quality Assurance