Quality Assurance with ETICS – multi-node automated testing

INFSO-RI-223782

Quality Assurance with ETICS – multi-node automated testing

CGW 09 M.Zurek, A. A. Rodriguez, A. Aimar, A. di Meglio, L. DiniCERN

Krakow, 12 October 2009

INFSO-RI-223782

2

Contents

• ETICS Overview

• Multi-Node Testing• Q & A

ETICS Testing Tools CGW09

INFSO-RI-223782

3

ETICS Overview


INFSO-RI-223782

The Problem

4ETICS Online tutorials - Module 01

Distributed Computing, Distributed Development

INFSO-RI-223782

The ETICS System

5


• Continuous software build, test and QA verification system• Easily extensible with additional plugins• Scheduled or on-demand build and test jobs• User computers or remote distributed infrastructures• Built-in connectors to distribute build and test jobs on different types of infrastructures from standard job management systems to the grid • Open repository of configuration metadata, packages and build, test and QA reports• Support for standard package management systems like YUM• It’s multi-platform and independent from any specific language, build or test tool

INFSO-RI-223782

Architecture

6

1

Build/TestArtefacts

ConfigurationWeb Service

Report/MetricsDB

ConfigurationDB

Execution Engines(Metronome, gLite, UNICORE, etc)

Command Line User Interface

RepositoryWeb Service

ETICS Infrastructure

Physical WorkerNodes

Web Portal

Virtual OS Images

CGW09ETICS Testing Tools

INFSO-RI-223782

The Configuration Web Client

7

ETICS Online tutorials - Module 01

INFSO-RI-223782

Command-line Client

8


INFSO-RI-223782

The Repository Web Application

9


INFSO-RI-223782

Examples of Plugins

10


Metrics Type Programming languages/

technologies

Tool ETICS Plugin

Complexity static Java

Python

Javancss JCcnPlugin

PyComplexityPlugin.py

Design quality static Java Jdepend JDependPlugin

N of potential bugs static C/C++

Python

Perl

PHP

Java

Flawfinder,

RATS

PMD

Findbugs

CFlawfinderPlugin

CPyPhpRatsPlugin

JPmdPlugin

JFindbugsPlugin

N of potential bugs dynamic C/C++ Valgrind CValgrindPlugin

Lines of code static All SLOCCount SLOCCountPlugin

Coverage dynamic Java Emma

Cobertura

JUnitemmaPlugin

JCoberturaPlugin

Unit tests success rate

dynamic Java

Python

JUnit

PyUnit

JUnitPlugin

JUnitreportsPlugin.py PyUnitPlugin.py

Compliance with standards

static IPv6

WSI

IPv6Plugin WSInteroperabilityPlugin

Profiling dynamic C/C++

Java

Jrat

Valgrind

JRatPlugin

CValgrindPlugin

INFSO-RI-223782

11

Multi Node


INFSO-RI-223782

12

Multi-Node

ETICS Testing Tools

The ability to setup scenarios where multiple services are automatically deployed in multiple nodes.

These services must be able to work properly as if they would be installed manually by a user performing any required operation in the required order.

CGW09

INFSO-RI-223782

13

ChallangeService Synchronization

NODE1

NODE2

WEBSERVERSERVICE

TESTSUITESSERVICE

CLIENTSERVICE

1

32

NODE1

NODE2

WEBSERVERSERVICE

CLIENT1SERVICE

1

2

NODE4

TESTSUITESSERVICE

3

NODE3

CLIENT2SERVICE

2


INFSO-RI-223782

14

ChallangeInformation Interchange

NODE1

TOMCAT1SERVICE

WEB APPLICATIONSERVICE

CONFIGURATIONPHASE

CONFIGURATIONPHASE

NODE2

MYSQL1SERVICE

INITPHASE

NODE2.MYSQL1.INIT.IP

MYSQL.INIT.IP

NODE1.TOMCAT1.CONFIG.CATALINA_HOME

TOMCAT.CONFIG.CATALINA_HOME

LOGGER = CONSOLE

RAM = 256MB THREAD = 300 CACHE = TRUE

REQUIRED (GLOBAL / NODE)

OPTIONAL(WITH DEFAULTS)

-

++

---

--

- + IN / OUT

= DESIGN-TIME


INFSO-RI-223782

15

ChallangeFlow Monitoring

NODE1

TOMCAT1SERVICE

WEB APPLICATIONSERVICE

CONFIGPHASE

CONFIGURATIONPHASE

NODE1.TOMCAT1.CONFIG.CATALINA_HOME

TOMCAT.CONFIG.CATALINA_HOME

RAM = 256MB THREAD = 300

LOG

NODE1: START

NODE1.TOMCAT1: START

NODE1.TOMCAT1.CONFIG: RAM = 256MB && THREAD = 300

NODE1.TOMCAT1.CONFIG: START

NODE1.TOMCAT1.CONFIG: ENDCATALINA_HOME = /usr/share/tomcat/

NODE1.TOMCAT: END

NODE1.WA: START

NODE1.WA.CONFIGURATION: STARTCATALINA_HOME = /usr/share/tomcat/

NODE1.WA: FAILED: ERROR MESSAGE

NODE1: END

ERROR

/usr/share/tomcat/


INFSO-RI-223782

16

The ETICS Solution

• ETICS-SET is used to publish key-value pairs or only keys to the central information system in order to make the information available to all the other nodes, services or phases.• ETICS-GET is used to get (blocking or not-blocking) the information published by some local or remote execution of ETICS-SET.• ETICS-LOG is used to log and for real-time monitoring.

ETICS Testing Tools

Server Node Client NodeCentral IS[ … INSTALL …]

[ … CONFIG … ]

[ … START … ]

ETICS-SET S.IP = x.x.x.x

ETICS-LOG “RUNNING”

ETICS-GET C.DONE

[… STOP AND EXIT …]

$IP = ETICS-GET S.IP

[ … USE $IP … ]

[… RUN TEST …]

ETICS-SET C.DONE

[… EXIT …]

S.IP = x.x.x.x

S: “RUNNING”

C.DONE

CGW09

INFSO-RI-223782

The PING Example (1/3)


INFSO-RI-223782



INFSO-RI-223782

Challenges

• Deployment automation in the context of complex software development is a prerequisite for automatic system testing

• Complexity of service configuration• Developers should provide deployable configurations, or

parameterised configurations

• External dependencies: all dependencies must be deployable automatically• Knowledge of these “external” components is not

necessarily with the end-user

• Privileged deployment/configuration operations• Root / Super User actions• Mix of normal user-level and root level actions

20


INFSO-RI-223782

Benefits

Automation of complex processes

Automatic setup of complex environment

Automatic generation of reports

Save on resources and time• More scenarios can be explored• Less human errors

Form the foundation for more advanced system tests

21


INFSO-RI-223782

Thanks!

22

http://www.eticsproject.eu


Quality Assurance with ETICS – multi-node automated testing

Documents

Transcript of Quality Assurance with ETICS – multi-node automated testing