Test case documentation and development for integration ... · Federates subscribing to a RPR 2...

CAN UNCLASSIFIED

CAN UNCLASSIFIED

Test case documentation and development for integration with NATO MSG-134 test suite

Dan Radulescu Laurence Olivier Marion-Ouellet Prepared By: OODA Technologies Inc. 4580 Circle Rd Montréal (Qc), H3W 1Y7 PWGCS Contract Number: W7707-145677 Contractor's date of publication: April 4, 2017 Technical Authority: Allan Gillis, Scientific Programmer

Terms of Release: This document is approved for Public release (unrestricted access to the document).

Defence Research and Development Canada

Contract Report

DRDC-RDDC-2017-C179

August 2017

CAN UNCLASSIFIED

CAN UNCLASSIFIED

IMPORTANT INFORMATIVE STATEMENTS

Disclaimer: This document is not published by the Editorial Office of Defence Research and Development Canada, an agency of the Department of National Defence of Canada, but is to be catalogued by DSTKIM, who manage the national repository for Defence S&T documents. Her Majesty the Queen in Right of Canada (Department of National Defence) makes no representations or warranties, express or implied, of any kind whatsoever, and assumes no liability for the accuracy, reliability, completeness, currency or usefulness of any information, product, process or material included in this document. Nothing in this document should be interpreted as an endorsement for the specific use of any tool, technique or process examined in it. Any reliance on, or use of, any information, product, process or material included in this document is at the sole risk of the person so using it or relying on it. Canada does not assume any liability in respect of any damages or losses arising out of or in connection with the use of, or reliance on, any information, product, process or material included in this document.

This document was reviewed for Controlled Goods by Defence Research and Development Canada using the Schedule to the Defence Production Act.

Template in use: (2010) CR EL1 Advanced Template_EN 2017-07-27-V01_WW - kag july 31.dotm

© Her Majesty the Queen in Right of Canada (Department of National Defence), 2017

© Sa Majesté la Reine en droit du Canada (Ministère de la Défense nationale), 2017

Test case documentation and development forintegration with NATO MSG-134 test suite

Dan RadulescuLaurence Olivier Marion-Ouellet

Prepared By: OODA Technologies Inc.4580 Circle RdMontreal (Qc), H3W 1Y7514-903-4747

Prepared For: Defence Research & Development Canada, Atlantic Research Centre9 Grove Street, PO Box 1012Dartmouth, NSB2Y 3Z7902-426-3100 ext.203

Technical Authority: Allan Gillis, Senior Scientific ProgrammerContract Number: W7707-145677Call Up Number: 16Project: Transmission validation of Entity Types, Identifiers and Weapon Fires through RTI.Report Delivery Date: April 4, 2017

The scientific or technical validity of this Contract Report is entirely the responsibility of thecontractor and the contents do not necessarily have the approval or endorsement of the Departmentof National Defence of Canada.

© Her Majesty the Queen in Right of Canada, as represented by the Minister of National Defence, 2017.

© Sa Majeste la Reine (en droit du Canada), telle que representee par le ministre de la Defense nationale,2017.

This page is intentionally left blank.

Contents

Contents 3

List of Figures 7

List of Tables 9

1 Project Scope 1

2 Abstract Test Cases 3

2.1 Abstract Test Case Description RPR Base Entity Validation . . . . . . . . . . . . 3

2.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.3 Test Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.4 The Stand-In SuT (SISuT) . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.5 The Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Abstract Test Case Description RPR Weapon Fire Interaction (WFI) Validation . 6

2.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.3 Test Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.4 The Stand-In SuT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.5 The Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 Software Design 9

3

Study Report RISOMIA Call-up 16

3.1 Base Entity related projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.1.1 Entity Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1.2 Entity Corruptor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1.3 Entity Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.1.4 Base Entity Test Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 WeaponFire related projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2.1 WeaponFire Corruptor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2.2 WeaponFire Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.2.3 WeaponFire Test Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4 Lessons Learned 15

4.1 The learning curve is steep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2 Automation is key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.3 Start grouping common classes classes early and use them . . . . . . . . . . . . . . 16

4.4 Dealing with Gradle circular dependencies . . . . . . . . . . . . . . . . . . . . . . . 17

4.5 Weapon Fires are precise structures . . . . . . . . . . . . . . . . . . . . . . . . . . 17

5 IVCT Suggestions 19

5.1 Use of exceptions in regular execution . . . . . . . . . . . . . . . . . . . . . . . . . 19

5.2 Expand the TS HelloWorld example . . . . . . . . . . . . . . . . . . . . . . . . . . 19

6 Way Ahead 21

Bibliography 23

Appendix A Install Guide A-1

A.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1

A.2 TC Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1

A.3 Entity Agent, Weapon Fire, Corruptors Installation . . . . . . . . . . . . . . . . . A-2

A.4 Running an Agent and TC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-2

4

The use or disclosure of the information on this sheet is subject to the restrictions on the title page of this document.

CONTENTS

A.5 Running a Corruptor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-3

5


OODA Technologies Inc.



6


LIST OF FIGURES

List of Figures

3.1 Base Entity Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.2 Weapon Fire Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

7





8


LIST OF TABLES

List of Tables

2.1 Entity Type attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 Entity Identifier attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.3 Federate Identifier attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.4 Weapon Fire Interaction Class Parameters . . . . . . . . . . . . . . . . . . . . . . . 7

9





10


LIST OF TABLES

List of Acronyms

ATC Abstract Test Case

CSV Comma Separated Value

DIS Distributed Interactive Simulation

EWG Enumerations Working Group

FAD Federation Agreement Document

FOM Federation Object Model

HLA High Level Architecture

IDE Integrated Development Environment

IVCT Integrated Verification & Certification Tools

JMS Java Message System

ORBAT Order of Battle

RPR Real Time Platform Reference FOM

RTI Run Time Infrastructure

SISO Simulation Interoperability Standards Organization

SISuT Stand In System under Test

SuT System under Test

TC TestCase

TS Test Suite

WFI Weapon Fire Interaction

11





12


Part 1

Project Scope

The Integrated Verification & Certification Tools (IVCT) were developed to support compliancetesting and certification of simulation components in a federation. In this call-up, the main concernis to expose errors a System under Test (SuT) could produce when publishing objects within aFederation. The assessments do not cover High Level Architecture (HLA) compliance but ratherfederation agreement conformity. In order to ensure a common understanding, all participantswithin a federation must agree on the enumeration makeup of simulated objects. Errors aretherefore any deviation from the common agreement, generally referred to as Federation AgreementDocument (FAD). These could be missing numbers, information that is erroneous or in a formatthat is not in a recognized standard. All development and tests were done in Pitch Run TimeInfrastructure (RTI).

This call-up is based on a previous work (Task 200 under W7714-08-3663) which had the responsi-bility of transmitting two objects through HLA: an Entity Identifier and an Entity Type. OODATechnologies leveraged this work in the development of a test case covering the Entity Identifierand Entity Type conformity with the FAD. These Real Time Platform Reference FOM (RPR)enumeration structures are mandatory attributes of the Base Entity object class. The developedtests therefore verify that a SuT publishes Base Entity objects with an identical makeup to thoseof the FAD.

A second part of this contract concerns the validation of Weapon Fire Interactions. Again, a testhas been developed to verify that its parameters are properly transmitted through a SuT to IVCT.Both tests (Base Entity and Weapon Fire) follow the same broad procedure:

1. A SuT or Stand In System under Test (SISuT) creates or joins the federation.

2. IVCT’s Test Suite (TS) joins the federation following the reception of a message throughActiveMQ. This message contains the location of the Federation Object Model (FOM), thetype of the test (Base Entity or Weapon Fire) and the FAD’s location.

3. The SuT broadcasts the testcase (Base Entity or Weapon Fire).

(a) The SuT reads its properties file to get the FOM and testcase path.

1


(b) The testcase CSV files are read and java objects are created from them.

(c) The Java objects are then converted to RPR FOM compliant objects, encoded andpublished through the HLA interface to the federation.

4. The HLA interface of IVCT’s Test Suite detects the newly created HLA objects and decodesthem into java objects.

5. The IVCT’s test suite reads the FAD and creates another set of Java objects.

6. The operator executes the validation test where objects discovered on the RTI are comparedagainst those of the FAD.

(a) Java objects discovered on the RTI are matched with FAD Java objects based on theiridentifiers. If a match is not found, the test grants a failure for this object.

(b) Matched objects then compare their respective attributes or parameters. If one or moreof these parameters do not match, the test grants a failure for this object.

(c) The whole testcase is then assigned a pass or fail grade. A pass is awarded only if eachjava object in the FAD has found a match and has passed comparison.

7. The test ends.

The work accomplished in this contract was to create the appropriate code to read, create, modify,broadcast, listen and compare Base Entities and Weapon Fire Interaction that are RPR compliant.The details for each of these steps are described in the chapters 2 and 3.

2


Part 2

Abstract Test Cases

2.1 Abstract Test Case Description RPR Base Entity Validation

2.1.1 Introduction

This test case is used to evaluate the fidelity of Entity Identifier and Entity Type enumerations ofFederates subscribing to a RPR 2 compliant Federation architecture. SISO-STD-001-2015 GRIMRPR FOM V2 ([1]) and SISO-REF-010-2015 ([2]) describe object classes, interactions and enu-merations respectively.

2.1.2 Background

RPR Entity Type enumerations are based on the Distributed Interactive Simulation (DIS) standarddescribed in IEEE-STD-1278-1([3]). The high level entity record structure is reproduced below(2.1).

“ Entity Types are arranged in a hierarchical order such that higher-fidelity simulations may depictdetailed representations of an entity (such as an F-16B) while lower-fidelity simulations may depictthe same entity in a more generic manner (such as an F-16 or a Fighter Aircraft).” [3]

The SISO Enumerations Working Group (EWG) has developed a document with specific numericalvalues as schema and repository for enumerations used in RPR. A few entity type records arereproduced below for reference:

1.1.225.1.1.1 M1 Abrams1.1.225.1.6.1 M41C1.1.225.2.41.1 LAV-2

The other attribute of a Base Entity is its Entity Identifier. Entity Identifiers are created accordingto the SISO-STD-001 document [2]. The HLA structure is reproduced below on table 2.2:

3


Table 2.1: Entity Type attributes

EntityType Record

Entity Kind 8 bit enum

Domain 8 bit enum

Country 16 bit enum

Category 8 bit enum

Sub Category 8 bit enum

Specific 8 bit enum

Extra 8 bit enum

Table 2.2: Entity Identifier attributes

EntityIdentifier Record

Federate Id FederateIdStruct (See table 2.3)

Entity Number UnsignedInteger16

Table 2.3: Federate Identifier attributes

FederateIdentifier Record

Site Id UnsignedInteger16

App Id UnsignedInteger16

4


Part 2. Abstract Test Cases

2.1.3 Test Environment

The TestCase (TC) must execute within the IVCT framework whereas the SuT can be any RPRFOM compliant Federate that needs to be certified. The SuT must create a Federation that IVCTwill then join in order to perform the actual tests.

2.1.4 The Stand-In SuT (SISuT)

A SISuT or dummy federate is available for the validation of the TC scenario. The stand-in em-ulates an Order of Battle (ORBAT) situation where a set of entities are generated as part of acommon federation execution. To achieve this, the stand-in SuT creates a series of BaseEntityobjects from a previously defined FAD. The FAD is a Comma Separated Value (CSV) file con-taining EntityIdentifier and EntityType dot-strings. It is constructed off of SISO-REF-010.csv. Autility project is also provided for the creation of the FAD with user controls for the number ofentities to generate. A second utility project modifies an existing FAD with errors to simulate aFederate producing incompatible enumerations. These corrupted FADs can be used to validate atest suite’s error detection accuracy. The SISuT logs the added errors for comparison with theTC’s validation.

2.1.5 The Test

The SuT is a federate supplying a list of Base Entities as part of an ORBAT. These Base Entitiesare composed of an Entity Identifier and Entity Type. The SuT is charged with starting a federationfor IVCT to join, and defines a set of entities that the TC can subscribe to. These can be anyRPR leaf entities that are transferable between federates. After subscribing to the federation, theTC establishes the RPR2 FOM compliance of the SuT entities and logs the results.

1. Create the FAD.

(a) Set the value of error mutation chance (if using the SISuT).

(b) Load the CSV files in the testcase folder.

2. Start the SuT to generate and publish a set of BaseEntity from the FAD.

3. Start the IVCT.

4. User specifies the TC to run within the test case parameters sent through ActiveMQ.

5. TC executes the test suite.

(a) The test suite joins the federation execution.

(b) The test suite subscribes to RPR Entity objects.

(c) The test suite evaluates the validity of each generated BaseEntity and records results.The validity includes strict comparison between FAD objects and RTI discovered ob-jects.

5




2.2 Abstract Test Case Description RPR Weapon Fire Interac-tion (WFI) Validation

2.2.1 Introduction

This test case is used to verify the integrity of WFIs of Federates publishing under a RPR 2compliant Federation architecture. SISO-STD-001-2015 GRIM RPR FOM V2 [1] describes theinteraction and its parameters.

2.2.2 Background

RPR2 WFIs are closely based on the FirePDU layout from section 5.3.4.1 of IEEE Std 1278.1[3]. The fields “FireControlSolutionRange”, “FireMissionIndex”, “QuantityFired”, “TargetObjec-tIdentifier”, “MunitionObjectIdentifier” and “RateOfFire” are optional. The other fields must beprovided values by the interaction sender. A full list of the parameters can be found at table 2.4.

On this table some structures need additional details; an EventIdentifierStruct is a structurecomposed of an EventCount UnsignedInteger16 and an IssuingObjectIdentifier Octet. A World-LocationStruct is a structure composed of three elements : x, y and z in UnsignedFloat64. AVelocityVectorStrut is a structure composed of three elements : x, y and z in UnsignedFloat32.The EntityTypeStruct has been previously described in detail in section 2.1.

2.2.3 Test Environment

The TC must execute within the IVCT framework whereas the SuT can be any RPR FOM com-pliant Federate that needs to be certified. The SuT must create a Federation that IVCT will thenjoin in order to perform the actual tests.

2.2.4 The Stand-In SuT

A SISuT or dummy federate is available for the validation of the TC scenario. The stand-increates a number of WFIs from a previously defined FAD. This FAD is a CSV file containingat least 13 columns for the 13 elements described in Table 2.4. Fields that contain a nestedsubstructure separate their elements with dots if they are integers (i.e. EventIdentifier = 123.45).If the internal elements are floats, then the separators are colons (i.e. WorldLocationStruct =123.4:567.89:123.456). The SISuT also contains a utility project to create a percentage of mutationsin a FAD for comparison validation purposes.

6


Part 2. Abstract Test Cases

Table 2.4: Weapon Fire Interaction Class Parameters

WeaponFire Interaction Class

Parameter Name Structure

EventIdentifier EventIdentifierStruct

FireControlSolutionRange Float32

FireMissionIndex UnsignedInteger32

FiringLocation WorldLocationStruct

FiringObjectIdentifier Octet

FuseType 16 bit enum

InitialVelocityVector VelocityVectorStruct

MunitionObjectIdentifier Octet

MunitionType EntityTypeStruct

QuantityFired UnsignedInteger16

TargetObjectIdentifier Octet

WarheadType 16 bit enum

7




2.2.5 The Test

The TC describes listeners to catch any incoming WFI. Since interactions are not permanent inPitch, it is imperative to start the TC before creating Weapon Fire Interactions. These interactionswill be created from the SuT and after reception by the TC, will be logged and compared to theFAD. Discrepancies between the initial FAD and the received Weapon Fires are highlighted.

1. Create the FAD.

(a) Set the value of error mutation chance (if using the SISuT).

(b) Load the CSV files in the testcase folder.

2. Start the SuT to generate and publish a set of WFI from the FAD.

3. Start the IVCT.

4. User specifies the TC to run within the test case parameters sent through ActiveMQ.

5. TC executes the test suite.

(a) The test suite joins the federation execution.

(b) The test suite subscribes to RPR WFIs.

(c) The test suite receives interactions.

(d) The test suite evaluates the validity of each generated WFI and records results. Thevalidity includes strict comparison between FAD interactions and interactions receivedthrough the RTI.

8


Part 3

Software Design

3.1 Base Entity related projects

According to the contract, both RPR Entity Type testing and Entity Identifier testing must beprovided. To answer to both these requirements, the base entity object from the RPR format hasbeen used because it includes an Entity Type and an Entity Identifier allowing us to simultaneouslycomplete both test cases. Four projects have been developed to create and validate the transmissionof base entities. The general use case of these is shown in figure 3.1.

To automate the creation of base entities, a project called Entity Generator has been created.Drawing from the SISO-REF-010-2015 entity type enumerations, the user enters the number ofentities he desires, N. N Entity Types will be drawn from the enumeration and be given randomlygenerated entity identifiers. The entities are then compiled in a CSV format output file, which nowincludes the “extra” field in Entity Type unlike the previous version from call-up W7719-135217.

To validate that the TS catches any poorly transmitted entities, errors must first be created inthe input files. To achieve this, the Entity Corruptor project reads a CSV formatted FAD, andmodifies it so x% of its enumeration fields will be changed, x being the percentage fixed by theuser.

The Entity Agent project SISuT or the actual SuT can now take a corrupted (or uncorrupted)CSV FAD to create and publish Base Entities to Pitch. In an operational setting where an actualSuT is being validated, the operator would create Base Entity objects likely by relying on thehuman readable Description field of the FAD.

Finally, the TS can subscribe to these entities, compare the received base entities and validatethem with the FAD.

9


Figure 3.1: Base Entity Use Case

3.1.1 Entity Generator

The Entity Generator works from the SISO-REF-010.CSV file as its raw input. This CSV filecontains the 9583 enumerated entity types and their description in SISO-REF-010-2015. It is usedas a reference to create valid entity types. An Index and an EntityId column are also presentin the source file but ultimately ignored. The generated EntityId is a dot separated String ofthree different numbers varying between 1 and 65534: SiteId, AppId and EntityNumber. The sameEntityNumber must not be generated twice as it is enforced to be a unique number according tothe RPR.

The project can be used by simply running without any input arguments where it will generate15 (default number) entities in a CSV file named “Default.csv”. A number can be given if theoperator wants a different quantity and the output filename can also be chosen.

Results are compiled in four columns : Index, EntityIdentifier, EntityType and Description with aheader for the first line.

3.1.2 Entity Corruptor

The goal of the Entity Corruptor project is to modify certain fields in a CSV formatted FAD togenerate a second CSV file which will serve as input for the Entity Agent if the operator wishesto introduce errors in the test. To do so, the operator specifies the output path for the corruptedCSV while the FAD file path is taken as input. He then enters the percentage of errors he wishes

10


Part 3. Software Design

to insert. The corruptor then calculates the number of errors to be added in the files in orderto respect that percentage. Randomly, the project assigns the fields to modify. A field is anynumber from the three EntityIdentifier components or from the seven enumerations in EntityType.The new number is randomly generated but made to respect the RPR guidelines. Finally, themodified CSV file is saved in the user specified location. The Description column is replaced witha CorruptionInformation column, describing which fields were modified for each base entity.

3.1.3 Entity Agent

The Entity Agent’s role is to first create or join the federation depending on its order of executionand then publish the list of entities described in the FAD (whether corrupted or not). To doso, it reads the content of the testcase folder, whose path is specified in the config.propertiesfile. The FOM’s path is also defined in the same file. Each user must make sure to modify theconfig.properties to their own setup.

If a test case analyst wishes to test the TS error detection, he must put corrupted CSV files in thetestcase folder instead of the original FAD. Alternatively, the config.properties can be modified topoint to the location of a corrupted FAD.

No matter the exact content of the testcase folder, the CSV files are then converted to Base EntityJava objects before being encoded into HLA and published to the Pitch RTI. The encoding fullycomplies with the GRIM RPR format. To achieve this, the SiteId and AppId are encoded in afederate identifier, which is encoded into the EntityId fixed record followed by the EntityNumber.The seven enumerations for the Entity Type are encoded into an EntityType fixed record. Finally,the BaseEntity Object is created, its attributes EntityId and EntityType are set and the object ispublished.

The exact process to create a Base Entity in HLA is to first make an object instance handle andupdate its attributes with a coder used on its ID dot String and type dot String. Finally, theEntity Instance is added to the controller’s instance map and through publishing, our Base Entityfinds its way to Pitch.

3.1.4 Base Entity Test Suite

The Test Suite, after connecting or creating the federation, has two ways of subscribing to theexisting base entities.

The first is the simplest, where the TS’s listener picks the base entities up as they are created bythe SuT/SISuT and decodes them. This method is only possible if the TS is already present whenthe publishing occurs. If the TS connects to an already existing federation where the Base Entitiesare already present, it must first request an update on each of an entity’s attributes.

Right after the TS detects the arrival of new BaseEntities, it decodes the HLA Objects and theirattributes into Java model objects and stores them in a map of discovered entities.

The test suite allows a 5 second delay between each Base Entity object discovery or attribute

11




reflection. When the time expires, the analyst is prompted to start the comparison test with thediscovered entities detected up to that point. He can run the comparison test by sending theappropriate menu command. For each discovered entity, an “equals” method is applied to checkwhether or not each FAD entity matches a discovered one. If every entity finds a perfect match,the test is considered passed. If an entity does not find a match or finds a match in EventId, whichis used as our key in our map, but one that does not contain the exact same fields (different kindsof numbers, for example), the test is considered as failed. The TS waits until every entity has beentested before declaring a pass or fail to allow every check to be made (a failure exception causesthe test to stop).

It is important to note that if the EntityId has been modified by the corruptor, the equivalentBase Entity will simply not be found since the Test Suite relies on EntityIds as key values. Insteadof giving a fail grade next to the object, the Test Suite will warn that this entity was not foundbut the overall result on IVCT will be a Fail, since not every base entity found its match.

3.2 WeaponFire related projects

A WeaponFire differs from a BaseEntity in many ways. Being an InteractionClass, a WeaponFirecontains parameters, not attributes, and does not have persistence on the RTI. Therefore, toensure the proper course of a test, it is important to activate the publishing of a WeaponFire onlywhen the TS WeaponFire has been started. To facilitate this, the WeaponFire Agent expects auser input before publishing. The main process between WeaponFire Agent and the TS is howeververy close to the one for base entities. The general use case of the Weapon Fire projects is shownin figure 3.2.

A WeaponFire is also written in a CSV file. These files, however, contain thirteen columns for alltheir parameters, which are previously described in their Abstract Test Case (ATC) description(section 2. Some columns contain multiple elements separated by a dot when these elements areintegers or separated by a colon when they are floats.

Just as in BaseEntity tests, a FAD corruptor project is also included to create errors in a FAD.

3.2.1 WeaponFire Corruptor

The WeaponFire Corruptor functions by reading a CSV weaponfire FAD, calculating the userspecified corruption level, and saving the resulting CSV file in the specified output. A WeaponFirepossesses a total of 24 fields :

• 2 for EventId

• 3 for a WorldLocationStruct

• 7 for the munition’s EntityTypeStruct

• 9 single value fields

12


Part 3. Software Design

Figure 3.2: Weapon Fire Use Case

The program randomly selects a number of fields on the available weaponfire entries and replacesthem with a randomly generated but RPR valid value. The now corrupted WFI are saved in thepreviously specified output folder, which can be called by the Agent and published to the RTI.

3.2.2 WeaponFire Agent

Since the InteractionClass is not persistent in Pitch, the actual publication of the interactions isput on hold until the analyst determines the test suite is ready to receive. At that point, theanalyst issues the publication order by pressing Y and Enter. The program then joins or createsthe federation, reads the testcases in the specified folder and finally encodes and publishes themin HLA. However, to ensure that they are received correctly, it is important to activate theWeaponFire Agent after the TS has been started. Once again, a config.properties file must be setup by the user before starting the agent. The path to the testcases and the FOM must be writtenin the appropriate fields. The testcase path can refer to a copy of the FAD or to a corruptedversion of it if the user wishes.

The WeaponFire Agent loads all CSV files in java WeaponFire objects containing the thirteenparameters before transforming those parameters in an HLA encoded value map. Lastly, this mapis sent as an interaction class through the WFI Class Handle and is published.

The main difference between Entity Agent and WeaponFire Agent is the complexity of the en-

13




coding. WeaponFire possesses more intricate structures as can be seen from the EventIdentifierparameter : an HLAfixedRecord containing an EventCount integer and an Issuing Object Identifierwhich is a RTIObjectId (HLAfixedRecord containing an HLAASCIIchar). Besides the structure ofthe encoding, the process remains the same.

3.2.3 WeaponFire Test Suite

The WeaponFire Test Suite only detects WFIs that are sent during its lifetime, unlike the BaseEn-tity Test Suite which can request an update and detect older base entities. The WeaponFire TestSuite, after its activation, listens to new WFIs created on the RTI and stores them and their pa-rameters in a WeaponFire Java Object map of discovered interactions. The Test Suite stores theseobjects until the test case analyst begins the comparison test by sending the menu command.

At this point, the CSV FAD is loaded and a comparison test begins for each WFI in the FAD.Since a unique Weapon Fire Event can only happen once, the Event Count, sub-parameter of EventIdentifier, has been chosen as the key value for the Weapon Fire Map as it should be unique. Foreach discovered WFI, the Test Suite tries to find a matching Event Count value in the FAD andthen checks if the values of every field match. Imprecise matches are accepted in the case of floatnumbers. For these, a margin of error of 1% is considered acceptable to account for roundingduring the encoding/decoding process.

If there is no matching Event Count in the FAD, the test fails. If there is a WFI with a matchingEvent Count but any of its content fields is different, the test fails. Only with a key and fullparameter match does the test grant a pass on the test. For the whole test to pass, every WFI inthe FAD needs to pass.

14


Part 4

Lessons Learned

4.1 The learning curve is steep

When it comes down to it, most of the code and logic behind each test case is simple but theintegration of a new test suite remains a challenge. A sizeable portion of obstacles are attributableto the number of different technologies and architectures incorporated in, and surrounding IVCT.Between HLA, Gradle, ActiveMQ, Pitch, RPR and IVCT, there is enough required knowledgeto consume a significant portion of a project’s resources. As such, the learning curve shouldbe factored into the planning phase of a call-up in order to avoid unexpected delays and usefuldocumentation should be logged for posterity. To that end, a short list of useful guides is presentedbelow.

• The Pitch HLA Tutorial is a well written and freely available introduction to HLA. It is awelcome reference for any HLA initiate regardless of the choice of implementation platform.

• The Pitch website also features links to a starter kit containing demo projects demonstratingfederate interactions in HLA. It can be a useful tool for developers in understanding howobjects, attributes and interactions work.

• Simulation Interoperability Standards Organization (SISO)’s digital library is the primarysource for any RPR related documentation. The most useful are SSIO-STF-001-2015GRIM RPR and SISO-REF-010 Enumerations for Simulation Interoperability.

• IEEE1278 documents are useful for legacy DIS support or when dealing with Entity Iden-tifier and Entity Type enumerations. These documents are available for purchase.

4.2 Automation is key

The large number of moving parts in a test suite project entails a need for build and test automa-tion. As reference, each test suite developed during this call-up relies on four distinct sub-projects

15


for its execution and tests (a CSV generator, a corruptor, an SISuT and the test suite itself).During a run, the test suite project needs to be triggered by ActiveMQ through Java MessageSystem (JMS) topics every run. Each one of these projects contains its own gradle build file aswell as a set of config files. Testing the productized code therefore implies calling four builds, andediting at least as many config files. Multiply that by the number of test suites and things quicklyget out of hand. Without automation, there is ample room for human error and lots of wastedtime debugging.

To reduce the potential for human error during tests, we have developed a testbed that combinesa chain of projects involved in a test suite including a JMS messenger that bypasses the needfor manual inputs in ActiveMQ. Developing a testbed is time consuming so its inclusion must becarefully weighed against its potential benefits.

Our TestBed only needs to be given three input arguments: the FAD location, the percentage ofcorruption to be inserted in the sent objects and the type of test (Base Entity or Weapon Fire).The program starts by loading the FAD and either copying it in a temp folder (If corruption is setto zero) or corrupting it and saving the result in the temp folder. This temp folder serves as therepository for the objects to be sent through HLA. The temp folder path is automatically givento the next part of the project, the Agent. The Agent usually starts by reading a config.propertiesfile, the path of which has to be specified. However, in order to streamline the process the Agent.java file itself was bypassed and its controller is instanced and given a EntityAgentConfig instancethat refers to the temp folder holding the FAD, corrupted or not.

The JMS TestRunner from TC/IVCT is then called and starts to listen/requests an update forthe entities. A JMS Messenger instance is then created and sends a command through ActiveMQto this TestRunner to set the FAD and test type. Once the user is ready, the comparison testcan begin. This test grants the testcase a pass or fail grade. A pass if the corruption level waszero should be the given grade. A fail if corruption was applied (since corrupted fields will notmatch those from the FAD). The final step in this TestBed would have been to automaticallycheck if the detected anomalies in the comparison test were the same that were introduced duringthe corruption process but this step would have required a greater level of effort than the projectallowed for. Due to this absence, the task remains the responsibility of the testcase analyst.

Streamlining the whole project builds would have been ideal but out of scope for this call-up. Ourefforts were limited to adding our test suites to the IVCT folder structure and build hierarchy andtherefore allowing for their compilation from the IVCT root build.

4.3 Start grouping common classes classes early and use them

As agents create essentially the same objects that the test suites discover, certain classes wereidentified as universal across several projects and grouped in a common library that could laterbe imported when needed. Unfortunately, as development progressed, new common code wasgenerated but omitted from the common library which resulted in different versions of what shouldhave been the same class spread across multiple projects. The ensuing mess and cleanup couldhave been avoided.

16


Part 4. Lessons Learned

While this lesson is mostly part of good code discipline, it is particularly pertinent here since thecall-up architecture is incumbent on common objects being created in separate java projects.

While it is perhaps acceptable that new code is not initially included in a common library, periodiccode review sessions should be used to identify these classes and move them as early as possible.

4.4 Dealing with Gradle circular dependencies

Circular dependencies arise from the new test suite extensions of the AbstractTestCase and relianceon IVCT’s HLA interfaces. The test suites require IVCT’s classes to compile and IVCT requiresthe test suites at runtime when their execution is called through the JMS topic.

Gradle cannot handle circular dependencies and there is unfortunately no way to fix the issue inthe scope of this call-up. To fix the problem, the classes in the TC project, common to the testsuite would have to be moved to a separate library (same as section 4.3).

In the meantime, we rely on a little bit of trickery to get Gradle to build. The TC project is builtseparately and then its libs are copied to the test suites. Each test suite then builds against theTC dependency in its local lib folder. Until the issue is fixed, for any change in the TC project tobe reflected in the test suites, care must be taken to rebuild the TC first and replace the compiledlibraries in each test suite before rebuilding the test suites themselves.

4.5 Weapon Fires are precise structures

While working on the physical entity generator, it was quickly noted how this project should betransferred to have its WeaponFire equivalent. However, while Base Entities are constructed from asequence of integers, WeaponFires contain many floats in their fields. Without intimate knowledgeof the SuT, it is hard to affirm that it can accept a very precise value for its WorldLocationStructor other non-optional Float structures and transmit it through Pitch.

While it is easy to enter an exact value in a CSV FAD, the simulator could very well not acceptan input with the same precision and cause problem. This possibility is maybe already covered bythe % of leniency to account for rounding errors in the encoding and decoding process. This valuecan easily be changed in the overridden method equals for WeaponFireImpl to a superior value ifneeded and such a solution is maybe sufficient for this problem.

17





18


Part 5

IVCT Suggestions

5.1 Use of exceptions in regular execution

The IVCT Abstract Test case class relies on Java exceptions to signal when a test case has failedor was inconclusive. The usual role for exceptions is to announce that the code has entered animproper state. One that may be recovered from if properly caught and handled but certainlya state that is not part of ordinary execution. However, in the context of testing, a test failureis simply one of the possible routine outcomes and not a malfunction of the IVCT and shouldtherefore not be handled by an exception.

A by-product of using exceptions in regular execution is that when a test fails, the IVCT haltsits execution and any of its subsequent tests. An early failure in a long list of Base Entities willtherefore prevent the testing of most entries.

Our solution to get around this problem is to test all discovered objects first, flag any failures butonly throw an exception at the very end. As a result, all entries are verified and logged regardlessof the verdict. Still, exceptions are not meant to be delayed until it’s convenient to throw them.That is not their appropriate use and may confuse future developers looking to expand the code.

It would be sufficient for the AbstractTestCase to rely on flags for test case failures and reservethe use of exceptions for faulty states such as missing files or parsing errors.

5.2 Expand the TS HelloWorld example

The TS HelloWorld example is a good starting point for test suite development but doesn’t demon-strate how integration with IVCT is supposed to take place. At runtime, the IVCT must call thetest suite and must therefore include its domain name in its classpath. While this is easily done inan Integrated Development Environment (IDE) such as Eclipse, it is a little less straightforwardin a command line environment.

Our solution was to integrate the new test suites in the IVCT build structure and set the runtime

19


dependencies in the TC gradle build. Building IVCT therefore sets the classpath dependencies forthe test suites so that they may be called at runtime.

The downside to this scheme is that all available test suites must be included in the IVCT distri-bution. Should the IVCT become large enough, an operator interested in a single test case mayfind the excess test suites to be a hindrance.

20


Part 6

Way Ahead

Looking forward, it’s critical that the test suites designed be tested against one or several SuT.Access to a vendor provided SuT was beyond the bounds of possibility for this call-up. It is alsoequally unlikely that the test suites will operate exactly as intended without testing on an actualSuT.

A main point of contention is likely to be that Base Entity objects may not normally be published bysimulators. Only more specific object classes such as Aircraft, Spacecraft, etc. may find themselveson the RTI. Our test suite would then be listening for object types that aren’t discoverable at all.The test suite would then have to be adapted to listen to all RPR publishable classes and extractthe Entity Identifier and Entity Type structs from them.

The Weapon Fire Interaction may also prove very difficult or time consuming for an operatorto generate in the exact terms expected by the FAD. Essentially, the FAD asks the analyst toproduce the exact situation that would generate the required interaction with all its specifiedfields. Depending on the federate’s controls, that may not be a possibility. In this case, it wouldbe possible to alter the comparison testing on the test suite to check for a series of enumerationvalues or ranged values in every field. The FAD would therefore have to specify a list of allowedvalues (or a range for numerical values) rather than the exact expected values.

21



22


BIBLIOGRAPHY

Bibliography

[1] Real time Platform Reference Federation Object Model Product Development Group.Siso-std-001-2015, standard for guidance, rationale and interoperability modalities for thereal-time platform reference federation object model version 2. Technical report, SISO, 2015.URL https://www.sisostds.org/DesktopModules/Bring2mind/DMX/Download.aspx?

Command=Core_Download&EntryId=30823&PortalId=0&TabId=105.

[2] Distributed Interactive Simulation Product Support Group Enumerations Working Group.Siso-ref-010-2015, enumerations for simulation interoperability, version 21. Technical report,SISO, 2015. URL https://www.sisostds.org/DesktopModules/Bring2mind/DMX/

Download.aspx?Command=Core_Download&EntryId=42916&PortalId=0&TabId=105.

[3] Institute for Simulation and Training. Ieee std 1278.1-1995, companion document,enumeration and bit encoded values for use with protocols for distributed interactivesimulation applications. Technical report, University of Central Florida, 2003. URLhttp://www.ist.ucf.edu/pdfs/Enum2003-for-IEEE-std-1278-1.pdf.

23



https://www.sisostds.org/DesktopModules/Bring2mind/DMX/Download.aspx?Command=Core_Download&EntryId=30823&PortalId=0&TabId=105




http://www.ist.ucf.edu/pdfs/Enum2003-for-IEEE-std-1278-1.pdf



24


Appendix A

Install Guide

A.1 Requirements

1. JDK 8 installed and JAVA HOME environment variable set to JDK root folder.

2. Ensure ActiveMQ is installed.

3. An RTI (e.g. PitchRTI) is installed.

4. An internet connection is required to download libraries.

A.2 TC Installation

1. Extract the IVCT Framework folder to the desired location.

2. Since we can’t distribute RTI libs, they have to be included in the classpath manually:

(a) Open IVCT_Framework/externallib.gradle in a text editor.

(b) Edit the ‘prtiLib’ value to point to the library folder of your local RTI library (Eg.prti1516e/lib/)1.

3. Open a terminal, navigate to the IVCT Framework folder and type each command in turnallowing for the task to complete every time:

(a) “gradlew build”.

(b) “gradlew installDist”.

1On a Windows machine, paths need to be typed with double backslash file separator. Eg. C:\\user\

\IVCTFramework\\TC\\...

A-1


A.3 Entity Agent, Weapon Fire, Corruptors Installation

Depending on which project you choose to run, ABC is replaced with the project name.

1. Extract the folder to the desired location.

2. Since we can’t distribute RTI libs, they may have to be included in the classpath manuallyfor the Agent projects:

(a) Open ABC/externallib.gradle in a text editor.

(b) Edit the ‘prtiLib’ value to point to the library folder of your local RTI library (Eg.prti1516e/lib/) (See previous footnote).

3. Edit the config file if the project is an Agent:

(a) Open ABC/resources/config/config.properties in a text editor.

(b) Ensure that the testcaseDir and fom variables point to the appropriate folders under/resources. Use absolute paths if necessary (See previous footnote).

4. Open a terminal, navigate to the ABC folder and type each command in turn allowing forthe task to complete every time:

(a) “gradlew build”.

(b) “gradlew installDist”.

Note that these projects rely on the compiled IVCT Common libraries to work, stored under the‘Shared’ folder. If the IVCT Common project is modified, the ‘Shared’ .jar folder will have to beupdated for the changes to take effect.

A.4 Running an Agent and TC

Depending on which Agent you choose to run, ABC is replaced with the Agent name.

1. Ensure that the RTI is running and ready to receive connections.

2. Also ensure that ActiveMQ is running.

3. The gradlew installDist command has produced .bat runnables under ABCAgent/build/

install/ABCAgent/bin/ and TC/build/install/TC/bin/, respectively. Run each of thesein a separate terminal. At this point, the EntityAgent will create and publish objects fromits build/install/EntityAgent/resources/testcases folder. The WeaponFireAgent onthe other hand will wait for the user to start the test suite before sending its interactions.The TC will create a ‘commands’ topic and will listen for incoming testcase commands.

4. Open a web page to the ActiveMQ manager (by default at “http://localhost:8161/”).

A-2


Appendix A. Install Guide

5. Under Topics, then ‘commands’, enter a JSON message of the following structure:

{

"commandType": "startTestCase"

"sequence": "1"

"tsRunFolder": ""

"testCaseId": "ca.drdc.be.TestMaster"

"tcParam":

{

"federationName" : "IVCT_Federation",

"rtiHostName" : "localhost",

"sutFederateName" : "BaseEntityGenerator",

"testcaseDir" : "<path_to_EntityAgent>/EntityAgent/resources/testcases",

"fom" : "<path_to_EntityAgent>/EntityAgent/resources/fom"

}

}

The testCaseId is currently set for a Base Entity Test but can be changed to “ca.drdc.wf.TestMaster”for a WeaponFire test.

Make sure to fill in the path to testcaseDir and FOM according to your local config. Useabsolute paths when in doubt. Again, double backslash file separator when on Windows.Send the message when ready.

6. When running the BaseEntityAgent, the TC should print out logs of reflected attributesas more BaseEntity objects are discovered. When all objects are discovered and processedlocally, a menu appears prompting the user to proceed with the test.

7. When running the WeaponFireAgent, the user should press ’y’ and Enter at this point tosend the interactions to the test suite. The user must then wait for all the interactions to bereceived by the test suite.

8. In both cases, press ‘1’ in the TC command line prompt to start the comparison test. At thispoint the TC will compare the RTI discovered objects with the ones read in the testcasesfolder and. Unless errors have been manually introduced, all lines should print “OK”.

A.5 Running a Corruptor

1. The gradlew installDist command has produced .bat runnables under ABCCorruptor/build/install/ABCCorruptor/bin/.

2. You can now run the Corruptor.

(a) Type ABCCorruptor.bat, the input .csv file path, the output .csv file path and aninteger representing the percentage of corruption separated by spaces. Once again, usedouble backslashes if working in Windows.

A-3



CAN UNCLASSIFIED

CAN UNCLASSIFIED

DOCUMENT CONTROL DATA (Security markings for the title, abstract and indexing annotation must be entered when the document is Classified or Designated)

1. ORIGINATOR (The name and address of the organization preparing the document.

Organizations for whom the document was prepared, e.g., Centre sponsoring a

contractor's report, or tasking agency, are entered in Section 8.)

OODA Technologies Inc. 4580 Circle Rd Montréal (Qc), H3W 1Y7 514-903-4747

2a. SECURITY MARKING (Overall security marking of the document including

special supplemental markings if applicable.)

CAN UNCLASSIFIED

2b. CONTROLLED GOODS

NON-CONTROLLED GOODS DMC A

3. TITLE (The complete document title as indicated on the title page. Its classification should be indicated by the appropriate abbreviation (S, C or U) in

parentheses after the title.)

Test case documentation and development for integration with NATO MSG-134 test suite

4. AUTHORS (last name, followed by initials – ranks, titles, etc., not to be used)

Radulescu, D.; Marion-Ouellet, L. O.

5. DATE OF PUBLICATION (Month and year of publication of document.)

August 2017

6a. NO. OF PAGES

(Total containing information,

including Annexes, Appendices,

etc.)

27

6b. NO. OF REFS

(Total cited in document.)

3

7. DESCRIPTIVE NOTES (The category of the document, e.g., technical report, technical note or memorandum. If appropriate, enter the type of report,

e.g., interim, progress, summary, annual or final. Give the inclusive dates when a specific reporting period is covered.)

Contract Report

8. SPONSORING ACTIVITY (The name of the department project office or laboratory sponsoring the research and development – include address.)

DRDC – Atlantic Research Centre Defence Research and Development Canada 9 Grove Street P.O. Box 1012 Dartmouth, Nova Scotia B2Y 3Z7 Canada

9a. PROJECT OR GRANT NO. (If appropriate, the applicable research

and development project or grant number under which the document

was written. Please specify whether project or grant.)

06AE

9b. CONTRACT NO. (If appropriate, the applicable number under

which the document was written.)

10a. ORIGINATOR’S DOCUMENT NUMBER (The official document

number by which the document is identified by the originating

activity. This number must be unique to this document.)

DRDC-RDDC-2017-C179

10b. OTHER DOCUMENT NO(s). (Any other numbers which may be

assigned this document either by the originator or by the sponsor.)

11a. FUTURE DISTRIBUTION (Any limitations on further dissemination of the document, other than those imposed by security classification.)

Public release

11b. FUTURE DISTRIBUTION OUTSIDE CANADA (Any limitations on further dissemination of the document, other than those imposed by security

classification.)

12. ANNOUNCEMENT OF BIBLIOGRAPHIC INFORMATION (Any limitation to the bibliographic announcement of this document. This will normally

correspond to the Document Availability (11). However, where further distribution (beyond the audience specified in (11) is possible, a wider

announcement audience may be selected.))

Public release

CAN UNCLASSIFIED

CAN UNCLASSIFIED

13. ABSTRACT (A brief and factual summary of the document. It may also appear elsewhere in the body of the document itself. It is highly desirable that

the abstract of classified documents be unclassified. Each paragraph of the abstract shall begin with an indication of the security classification of the

information in the paragraph (unless the document itself is unclassified) represented as (S), (C), (R), or (U). It is not necessary to include here abstracts in

both official languages unless the text is bilingual.)

---------------------------------------------------------------------------------------------------------------

14. KEYWORDS, DESCRIPTORS or IDENTIFIERS (Technically meaningful terms or short phrases that characterize a document and could be helpful

in cataloguing the document. They should be selected so that no security classification is required. Identifiers, such as equipment model designation,

trade name, military project code name, geographic location may also be included. If possible keywords should be selected from a published thesaurus,

e.g., Thesaurus of Engineering and Scientific Terms (TEST) and that thesaurus identified. If it is not possible to select indexing terms which are

Unclassified, the classification of each should be indicated as with the title.)

HLA; Simulation; Verification; NATO; Testing

Test case documentation and development for integration ... · Federates subscribing to a RPR 2...

Documents

Transcript of Test case documentation and development for integration ... · Federates subscribing to a RPR 2...