Tools for Test-Driven Product Modeling

Tools for Test-Driven Product Modeling

Tim Geisler Christophe Faure

webXcerpt Software GmbH Fysbee SAS

[email protected] [email protected]

CWG 2012, Berlin 2012-05-08

mailto:[email protected]

Context: ConfigModeler @

StandardizedStandardizedproductproductdescriptionsdescriptions



FormalizedFormalizedproductproductdescriptionsdescriptions



FormalizedFormalizedproductproductdescriptionsdescriptions ConfigModeler + VClipse

Eclipse-based IDEscode generation

CML asspecification and

documentation language

CWG 2010 ViennaCWG 2011 Cologne

ConfigModeler & VClipse @

RFC

Editor/IDE

Editor/IDE

ConfigModeler

VClipse

CML

VCML

Code Generation

Retrospective

● Product-model quality and modeler productivity increased by use of ConfigModeler.

− 2/3 of all live models made with ConfigModeler

● But: Functional testing by business showed that model implementation is not always what was intended by business.

Problems with Testing

● Product models are not tested− Acceptance tests by business

● Product models are not tested systematically− Test plans

● Product models are not tested automatically− Automated testing tools:

Fysbee ConfigScan, Sybit Model Tester

● Product models are tested too late− ???

Traditional Development and Testing Process

Design

Implement

Test

Test Earlier

Design

Implement

Test

Test-Driven Development

Design

Implement

Test

Test


Design

Implement

Test Test


Design

Implement

Test TestRefactor /Clean Up

Requirements

● Test cases should be written as soon as possible – even during the specification phase of the model.

● Certain test cases must be written by the modeler (together with the business?)

● Test cases must be executed quickly.● No media or tool changes for writing test cases

and executing tests.

Example: PC Configuratordeveloped with

Test-Driven Product Modeling(Screencast)

Developer's Tools

Developer's Tools

● rich test-language constructs

− values, BOM, status, domains, visibility

● test engine tests „real thing“− VC and IPC, on real system

● integrated into modeling IDE− navigation

● fast edit-and-test cycles:− model setup in SAP with one click

− test execution from inside modeling environment

● refactoring support− simultaneous maintenance of model and test cases

Test Engine ConfigConfigScanScan

● For VC and IPC● Test-case database inside SAP● Rich test language● RFC-enabled API

− Call test engine from external modeling environment − Upload test cases as XML files to ConfigScan's

test-case database for recurring test runs.− Execute XML streams as test cases with

ConfigScan's test engine with or without storing them – even on remote systems

ConfigConfigScanScan's Test Language

● XML-based sequential language

● Simulate user's interaction with the product model

● Commands on cstic level

− Set a value for a cstic

− Verify that values are (not) in the cstics domain

− Verify whether the cstic is (not) set to a specific value

− Verify that a cstic is set, unset, visible, hidden, set by default, …

● Commands on BOM item level

− Verify the quantity of a bom item

− Verify that a BOM item is complete, consistent, instantiated

● Hierarchical nesting of test groups

Test Language CMLT

● Integrated with CML

● All ConfigScan test statements

● Hierarchical models through „context“

● Nested test groups

● New compound statement „bom“

IDE for Editing CMLTtooltips

refactoring

templates

outlinequickfix

folding

content assist

navigation

Test Coverage

● „Static“ test coverage: which characteristics and materials are covered by test cases?

● Could be refined, e.g.:− Which value is covered by test cases?− Which dynamic domain is covered by test cases?

Integrating End-User Tests

● Model owner does acceptance teststypically in end-user environment: IPC

● Export configuration from IPC as .cfg file● Automatically transform .cfg file into CMLT test

cases− Represent only cstic settings, BOM, and status− No domain and user interaction sequence

● When defects are found, create small unit tests which reproduce failures.

Test Execution from ConfigModeler

● Execute ConfigScan simultaneously onmultiple files and multiple systems

● Additional parameters for ConfigScan● Store launch

configuration

ConfigConfigScanScan View – Display Logs

● ConfigScan logs are displayed in a tree view

● Various actions:

− Expand / collapse

− Next error

− Filter failed tests

− Save / load log

− History

− Re-run (failed) tests

ConfigConfigScanScan View – Navigate to Test

ConfigConfigScanScan View – Group by Error

● Failures can be grouped by the error message.

● This enables quick problem analysis when one defect is detected by several failed test cases.

VClipse & ConfigConfigScanScan

● VClipse: Eclipse-based modeling environment for the Variant Configurator, textual language VCML

● VClipse is backend for ConfigModeler

● ConfigScan integration is part of VClipse

● The presented interface from ConfigModeler is just an instantiation of the VClipse / ConfigScan integration.

● VClipse is open-source (www.vclipse.org)

− You can get the ConfigScan integration for free− Requirement: ConfigScan must be installed in your

landscape

VCML and VCMLT

Testing language VCMLT for VCML models(contributed to VClipse by Winfried Kung)

Adapt ConfigConfigScanScan Integration for Your Testing Language

● Create your own modeling language● Create your own testing language● You can use the ConfigScan integration by

implementing some simple interfaces− Translation of your testing language to ConfigScan

XML language− Optionally own styling code for tree view

● Example:VCMLT implementation on www.vclipse.org

Testing with ConfigConfigScan Scan (Test Type)

Testing with ConfigConfigScan Scan (Role/Type)

Testing With ConfigConfigScanScan

● From the beginning ConfigScan was designed for Test Driven Modeling (CWG Salzburg 2007)

● Test engines provide external API and detailed XML testing language

● Test are organized in standard SAP Catalogs● Test can be executed in standard SAP JOB

− Scheduled, periodic, …

● Dedicated mail notification of testing JOB● Test logs are hierarchic and stored in SAP LOG

Test Management in ConfigConfigScanScanTesting framework with several components● Test Engines (local/remote IPC and VC)

− Dynamic testing language

● Test Editor/Importer (material centric)

− Test document with nested test group

● Test Cockpit (catalog centric)

− Organize tests in SAP Catalog

− Schedule testing job

− Mail notification

● Job Monitor (job centric)

− Manage testing job

Testing with ConfigConfigScan Scan (Architecture)

Comparison withSoftware Engineering

Xunit test frameworks● Test language + test-execution framework

● Integrated into development environment

➔ VClipse / ConfigModeler - ConfigScan integration

Continuous Integration● System which continuously builds code

● Test cases are run periodically

● Notifications, if tests fail

● CI monitoring

➔ ConfigScan's test management

Experiences and Uses at NSN

● Used also for model migrationfrom legacy configurators

− Model behavior of legacy configurator as reference for test cases

● Scenarios during product modeling:− Development tests (D)− Tests for product transport (D Q P)− Tests for traceability (P with saved logs)

● Regression testing

Summary

● Test-driven product modeling is feasible using an automated test engine and an external modeling environment with test-engine integration.

● The presented methodology and tools have been in use at Nokia Siemens Networks in production for selected products since July 2011.

● Integrating different products from CWG members with open interfaces can increase customer value.

Tools for Test-Driven Product Modeling

Software

Transcript of Tools for Test-Driven Product Modeling