GDMX Distribution Service

Installing and Configuring

2 | P a g e

COPYRIGHT

All content of this document is copyright of DataGenic Ltd, unless explicitly stated otherwise. No part

of this document may be reproduced or transmitted in any form or by any means, electronic,

mechanical, photocopying, recording or otherwise, unless otherwise stated, without the prior

consent DataGenic Ltd.

Copyright © 2016 DataGenic Ltd. All rights reserved.

VERSIONING

Date Version Author Changes 1.0 13-07-2009 David Thomson Initial version (version 1.3)

1.1 31-05-2011 David Thomson Added features (version 1.6.0.3)

1.2 17-02-2012 Andrew Marsland Replaced/merged existing documentation (version 1.6.0.5)

1.3 15-03-2012 Andrew Marsland Added overview diagrams, usage scenarios

1.4 17-05-2012 David Thomson Added information on “GdmUseKerberosAuthentication” setting (version 1.6.0.8)

1.5 17-12-2012 Andrew Marsland Version 1.7.0.0 changes, compressed files supported in data sources and distribution config, added upgrade notes

1.6 24-06-2013 David Thomson Version 1.7.1.0 changes to handle collection from multiple FTP locations.

1.7 16-07-2013 David Thomson Modified the required .Net framework to be version 3.5.

1.8 19-05-2015 David Thomson Updated with new location for configuration files.

1.9 10-08-2016 David Thomson Updated the required .Net Framework to be version 4.0 plus the link to download.

2.0 22-08-2016 David Thomson Updated FTP and SFTP hostname from ftp.datagenic.net to include both ftp1.datagenic.net and ftp2.datagenic.net. Updated references to configuration file names App.config and log4net.config.

2.1 07-09-2016 David Thomson Added note on “Unblock” action required when Windows is stopping service from running. Added prerequisite C++ 2010 redistributable.

3 | P a g e

Contents 1 Overview ......................................................................................................................................... 4

2 Software Requirements .................................................................................................................. 4

3 Network Requirements ................................................................................................................... 5

3.1 Outbound ................................................................................................................................ 5

3.2 Proxy Support .......................................................................................................................... 5

4 Security Recommendations ............................................................................................................ 6

4.1 Running with a Service Account .............................................................................................. 6

4.2 Allowing for SPI or other packet inspection............................................................................ 6

5 Deployment Scenarios .................................................................................................................... 7

5.1 Proxy Servers ........................................................................................................................... 8

5.2 DMZ Delivery (2-step delivery) ............................................................................................... 9

5.3 Distributing to multiple GDM Servers (chaining) .................................................................. 10

6 Installation .................................................................................................................................... 11

6.1 Directory Listing .................................................................................................................... 11

6.2 Installation Steps ................................................................................................................... 11

6.3 Installation Notes .................................................................................................................. 12

6.4 Uninstalling the Service ........................................................................................................ 12

6.5 Upgrading from an earlier version ........................................................................................ 12

7 Service Configuration .................................................................................................................... 13

7.1 GdmxDistributionService.exe.config ..................................................................................... 14

7.2 DataSource-config.xml .......................................................................................................... 20

7.3 Distribution-config.xml ......................................................................................................... 22

8 Logging Configuration ................................................................................................................... 24

9 Appendix A – File & Directory Listing ............................................................................................ 25

10 Appendix B – Troubleshooting .................................................................................................. 27

10.1 Have all downloaded files been delivered? .......................................................................... 27

10.2 Has ECB_FX_20080606120030.xml delivered?..................................................................... 27

10.3 What GDMX files have been delivered on 12th November 2008? ....................................... 27

10.4 How to determine if there are failed deliveries? .................................................................. 27

10.5 How to find logs of activity of the service? ........................................................................... 28

10.6 How to Start / Stop the GDMX Distribution Service ............................................................. 28

4 | P a g e

1 Overview

The GDMX Distribution Service is a windows-based program which facilitates the downloading of

GDMX data files from the ‘DataHub’ (via DataGenic’s FTP site) and distributing the data files to GDM

servers. Data Sources to be collected and distribution of those data sources can be configured. It

runs as a Windows Service, so is easily accessible via Services in Windows’ Control Panel.

GDMX data files can be collected from a single location by the service, which might be a network

share, local directory or an FTP site. These data files get copied to a temporary folder (“Inbox”).

There is an attempt to deliver each of these files to all delivery locations (typically a GDM Server). If a

GDMX data file is delivered to all locations successfully, then it gets copied to the “Inbox\Delivered”

directory. If not, it remains in the Inbox so the service can attempt to re-deliver the file.

As well as distributing files to GDM Servers, GDMX Distribution must report back to the DataHub on

a successful delivery and loading of the data. This is usually a single HTTP call to a web service

located in the DataHub.

2 Software Requirements

The .Net Framework 4.0 and C++ 2010 Redistributable are the only prerequisites and so the service

should work where this framework and C++ runtime can be installed. The GDMX service has been

tested on Windows Server 2003, 2003 R2, 2008 and 2008 R2.

The runtime for the .Net Framework can be obtained from Microsoft at the following URL:

https://www.microsoft.com/en-us/download/details.aspx?id=17851

The runtime for C++ can be found at:

https://www.microsoft.com/en-us/download/details.aspx?id=14632 (x64)

https://www.microsoft.com/en-us/download/details.aspx?id=5555 (x86)

5 | P a g e

3 Network Requirements

3.1 Outbound

The service currently uses FTP or SFTP to contact the DataHub; this makes use of the following ports:

TCP 21 – FTP ftp1.datagenic.net

TCP 21 – FTP ftp2.datagenic.net

TCP 22 – SFTP ftp1.datagenic.net

TCP 22 – SFTP ftp2.datagenic.net

FTP is the default option, SFTP is not required, but is available should you prefer a more secure

connection.

The service also needs to connect to the DataHub to report on whether or not data deliver/loading

has been successful:

TCP 8080 – HTTP gdmxreporting.datagenic.net

TCP 443 – HTTPS gdmxreporting.datagenic.net

HTTP is the default option, HTTPS is not required, but again this is an option for customers that

prefer a secure HTTP connection.

Note: There are no inbound requirements for the GDMX Distribution Service.

3.2 Proxy Support

The GDMX Distribution service does support connecting to the DataHub via a proxy server for both

the FTP transfers and the HTTP reporting. This is discussed further in the configuration section. If

your proxy server uses a particular port then this will need to be opened for outbound on your

firewall.

6 | P a g e

4 Security Recommendations

4.1 Running with a Service Account

As with many Windows services; you may want to set up a separate Windows user account to run

the service. This is not required, but the service will install using the built-in ‘Network Service’

account. If you want greater control over access permissions then the appropriate permissions to

read/write should be applied to the installation folder. By default, all log files and GDMX data files

(temporary storage) are going to be stored on the same server.

If you do this, you will need to either add the windows user to the GDM Server (in the format

user@domain.internal) or update the GdmUsername property in the main configuration file.

4.2 Allowing for SPI or other packet inspection

Naturally if you wish to implement some form of packet inspection at a firewall or other border

device, you will not be able to use SFTP/HTTPS with the GDMX Distribution service. Please use the

default FTP/HTTP settings.

7 | P a g e

5 Deployment Scenarios

The typical GDMX Distribution deployment connects a server on a remote site to the DataHub

directly.

DataHub

INTERNET

Site Firewall

gdmxreporting.datagenic.net

ftp.datagenic.netDataGenic FirewallApplication Server

running

GDMX Distribution

On-site

GDM Server

This offers the simplicity of only having a single distribution point and is typically used in smaller

environments. The GDMX Distribution service can even run on the same machine as the GDM Server

as there are no conflicting ports. It is shown as 2 separate servers in these diagrams.

8 | P a g e

5.1 Proxy Servers In some organizations, the GDMX Distribution service may be configured to connect using a proxy

server. This is a supported configuration. The GDMX Distribution service has been tested with

SOCKS-compatible and standard proxies:

Proxy Method 1 (SITE site) USER ProxyUsername PASS ProxyPassword SITE Hostname USER Username PASS Password

Proxy Method 2 (USER user@site) USER Username@Hostname:Port PASS Password

Proxy Method 3 (USER with login) USER ProxyUsername PASS ProxyPassword USER Username@Hostname:Port PASS Password

Proxy Method 4 (USER/PASS/ACCT) USER Username@Hostname:Port ProxyUsername PASS Password ACCT ProxyPassword

Proxy Method 5 (OPEN site) USER ProxyUsername PASS ProxyPassword OPEN Hostname USER Username PASS Password

Proxy Method 6 (firewallId@site) USER ProxyUsername@Hostname USER Username PASS Password

Proxy Method 7 USER ProxyUsername USER ProxyPassword SITE Hostname:Port USER Username PASS Password

Proxy Method 8 USER Username@ProxyUsername@Hostname PASS Password@ProxyPassword

The configuration option for this can be specified in the main config file.

9 | P a g e

5.2 DMZ Delivery (2-step delivery) In larger organizations there will almost certainly be restricted firewall access for machines on the

internal network. The DataHub is an internet-based service and so normally these larger

organizations opt to allow a DMZ host (between their external perimeter firewall and internal

network firewall) to connect instead of connecting directly. This can be seen below:

DMZ

External Firewall

Proxy

Application Server

running

GDM Server

DMZ FTP DMZ File Share (smb)

Application Server

running

GDMX Distribution

Internal Firewall

From a data delivery point of view DataGenic needs to ensure that your data has been delivered

successfully, so the service would be installed on the internal network (along with the GDM Server).

There are 2 options:

1. Use a second instance of the GDMX Distribution service from within the DMZ

2. Use a custom FTP client or script to collect the data from the DataHub and then copy to a location visible by the internal GDMX Distribution service

To facilitate either of these options we can also make use of the GDMX Distribution service’s ability

to deliver to a network file share as an alternative to FTP. This is particularly useful if your

organization does not have a DMZ-based FTP server, which would allow access to the files internally

(see Distribution-config.xml for an example). Depending on how much control your organization

requires over the incoming files you may opt for either option 1 or option 2.

10 | P a g e

5.3 Distributing to multiple GDM Servers (chaining) There may be an instance where you require one data ‘feed’ from the DataHub to be delivered to 2

internal GDM Servers. This process is called “chaining”.

Application Server

running

GDM Server

Application Server

running

GDM Server

single DataHub FTP account

Application Server

running

GDMX Distribution

Firewall

This is particularly useful if you have only been provided a single FTP account to access the DataHub.

This can be configured in the Distribution configuration file by adding an additional distribution

point.

NOTE: Normally for DR or Test GDM servers an additional FTP account would be provided.

11 | P a g e

6 Installation

The GDMX Distribution service can be installed virtually anywhere on the server but we recommend

using:

C:\Program Files\DataGenic\GdmxDistributionService

You should allow for suitable growth, particularly with the Logs directory. This will vary based on the

amount of data you collect as well as the frequency. Please see the Logging Configuration for more

information. If required the service can be uninstalled, the folder moved and then reinstalled at the

new location.

6.1 Directory Listing

The GdmxDistributionService.zip file contains a directory ‘GdmxDistributionService’ containing the

following folders:

‘bin’ directory with application files

‘config’ directory with configuration files (since version 1.7.4.1)

‘install’ directory with service install/uninstall files

‘Logs’ directory which contains the service install logs and service runtime logs

6.2 Installation Steps

The installation steps are as follows:

1. Extract the contents of the GdmxDistributionService.zip in to a directory on server, e.g.

‘C:\Program Files\DataGenic’.

2. Open a command prompt (elevated using ‘Run As Administrator’ for Windows Vista/2008

onwards).

3. Navigate to the install directory:

cd “c:\Program Files\DataGenic\GDMXDistribution\install

4. Run ‘install.bat’

The installation will attempt to start the service, but there are a number of configuration options

which will need to be adjusted before this can be started successfully.

Possible Issues

If you happen to get the error message “HRESULT: 0x80131515” when installing, it’s likely due to

being this service installed on a version of Windows that protects files downloaded from the internet

(where the install was downloaded from the FTP site). Resolution for this is to open properties on

the file GdmxDistributionService.exe, and click the “Unblock” button at bottom of the General tab.

12 | P a g e

6.3 Installation Notes By default the service is named “DataGenic Gdmx Distributor” in the Control Panel -> Services, but

this can be changed and multiple services can be installed on same server by running

“InstallGdmxDistributorService <service_name>” in a command window.

NOTE: ensure that you do not have more than one service loading GDMX files from the same place

or to the same GDM server

If you do not need to use network shares for delivery or collection you can select the option to Log

On as Local System account. Otherwise you may need to configure with a different account. Ensure

this is an account which has permissions to any network shares that are used as part of the delivery

process.

6.4 Uninstalling the Service

The ‘uninstall.bat’ can be used to remove the service.

6.5 Upgrading from an earlier version When upgrading the GDMX Distribution service, you will need to overwrite the files in the ‘bin’

directory with the new version. Typically this does not require the service to be removed and re-

installed.

Please ensure you take a copy of the following files as they will be overwritten with the default

template files from the new release:

GdmxDistributionService.exe.config

DataSource-config.xml

Distribution-config.xml

If upgrading to version 1.7.4.1 or newer, you will need to move / rename configuration files. The following is required:

Move DataSource-config.xml to the config directory.

Move Distribution-config.xml to the config directory.

Move GdmxDistributionService.exe.config to the config directory and rename as app.config.

Move GdmxDistributionService.exe.log4net to the config directory and rename as log4net.config.

13 | P a g e

7 Service Configuration

There are 3 main configuration files that need to be adjusted:

App.config (called \bin\GdmxDistributionService.exe.config before 1.7.4.1)

DataSource-config.xml

Distribution-config.xml

The main application configuration file, ‘app.config’, contains the general settings that apply to all

GDMX delivery aspects. It’s an XML formatted file which can be easily edited.

The file ‘DataSource-config.xml’ contains the list of data sources that are required to be collected.

The file ‘Distribution-config.xml’ contains the list of data sources that are required to be collected

and the servers to which they must be delivered.

Each file is shown and explained in the following sections.

14 | P a g e

7.1 App.config <?xml version="1.0" encoding="utf-8"?>

<configuration>

<configSections>

<sectionGroup name="applicationSettings" type="System.Configuration.ApplicationSettingsGroup,

System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">

<section name="GdmxDistributionService.Properties.Settings"

type="System.Configuration.ClientSettingsSection, System, Version=2.0.0.0, Culture=neutral,

PublicKeyToken=b77a5c561934e089" requirePermission="false" />

</sectionGroup>

<section name="collectionPaths"

type="GdmxDistributionService.Configuration.CollectionSection,GdmxDistributionService" /> <section name="gdmWebServices"

type="DataGenic.DataManager.Config.ConnectionSection,DataGenic.DataManager" />

</configSections>

<collectionPaths>

<paths>

<add name="Primary" path="ftp://ftp1.datagenic.net/outbound" userName="user1"

password="password1" />

<add name="Secondary" path="ftp://ftp2.datagenic.net/outbound" userName="user2"

password="password2" />

</paths>

</collectionPaths>

<gdmWebServices>

<urls>

<add serverVersion="v2"

settingsServiceUrl="gdmWS/ConfigurationActionsService"

gdmViewServiceUrl="gdmWS/ProfileActionsService"

importExportServiceUrl="gdmxWS/GdmxActionsService"

modelServiceUrl="gdmWS/ProfileActionsService"

serverAdminServiceUrl="gdmWS/ProfileActionsService"

templateServiceUrl="gdmWS/ProfileActionsService"

/>

<add serverVersion=""

settingsServiceUrl="gdmWS/ConfigurationActionsService"

gdmViewServiceUrl="gdm/GdmViewActionsService"

importExportServiceUrl="gdm/ImportExportActionsService"

modelServiceUrl="gdm/ModelActionsService"

serverAdminServiceUrl="gdm/ServerAdminActionsService"

templateServiceUrl="gdm/TemplateActionsService"

/>

</urls>

</gdmWebServices>

<appSettings>

<add key="GdmxDocumentPath" value="" />

<add key="GdmxDocumentUserName" value="user" />

<add key="GdmxDocumentPassword" value="password" />

<add key="GdmxDocumentProxyAddress" value="" />

<add key="GdmxDocumentProxyUserName" value="" />

<add key="GdmxDocumentProxyPassword" value="" />

<add key="GdmxDocumentProxyMethod" value="" />

<add key="SocksHostname" value="" />

<add key="SocksPort" value="" />

<add key="SocksUsername" value="" />

<add key="SocksPassword" value="" />

<add key="SocksVersion" value="" />

<add key="GdmUserName" value="" />

<add key="GdmPassword" value="" />

<add key="InboxDirectory" value="" />

<add key="LogsDirectory" value="" />

<add key="DisableArchiving" value="false" />

<add key="DeliveryLogRecycleInterval" value="30" />

<add key="Interval" value="10" />

<add key="MaxFileSizeKb" value="0" />

<add key="MaxFileStorageDir" value="2000" />

<add key="NextFileWaitSeconds" value="10" />

<add key="SplitFilesAfterFail" value="false" />

<add key="TimeoutSeconds" value="7200" />

<add key="ClientName" value="ClientABC" />

<add key="EnableSourceFileDelete" value="true" />

<add key="DeleteSourceFileByWebService" value="true" />

<add key="EnableReporting" value="true" />

<add key="ReportByWebService" value="true" />

<add key="ReportLogPath" value="" />

<add key="WebProxyAddress" value="" />

<add key="WebProxyUserName" value="" />

<add key="WebProxyPassword" value="" />

</appSettings>

<applicationSettings>

<!--

<GdmxDistributionService.Properties.Settings>

<setting name="GdmxDistributionService_ProxyServices_ProxyWebService" serializeAs="String">

<value> https://gdmxreporting.datagenic.net:443/GdmxWeb/Services/ProxyWebService.asmx</value>

</setting>

15 | P a g e

</GdmxDistributionService.Properties.Settings>

-->

<GdmxDistributionService.Properties.Settings>

<setting name="GdmxDistributionService_ProxyServices_ProxyWebService"

serializeAs="String">

<value>http://gdmxreporting.datagenic.net:8080/GdmxWeb/Services/ProxyWebService.asmx</value>

</setting>

</GdmxDistributionService.Properties.Settings>

</applicationSettings></configuration>

An explanation of the configurable <appSettings> can be found below:

GdmxDocumentPath This is the FTP, share or local directory from where the

GDMX files will be collected. FTP must be in the format:

ftp://<username>:<password>@<ftp directory path>

DataGenic will provide this when connecting to the

DataHub.

File shares in the format \\<machine name>\<share

directory path>

If using a file share, the user running the service must have

permissions to access the share (see Running as a service).

For services which collect data from the DataGenic

DataHub, it is preferred to use the new settings

“collectionPaths”, added in version 1..7.1.0, as this allows

for service to collected files from a backup FTP server

should the main FTP server be out of service. See section

below.

GdmxDocumentUserName If not given in the GdmxDocumentPath this is the Username

of the remote service you are connecting to.

GdmxDocumentPassword If not given in the GdmxDocumentPath this is the Password

of the remote service you are connecting to.

GdmxDocumentProxyAddress Address of HTTP proxy server (if using a proxy server to

access the GdmxDocumentPath).

GdmxDocumentProxyUserName Proxy server username.

GdmxDocumentProxyPassword Proxy server password.

GdmxDocumentProxyMethod Selects the proxy method to use (see Proxy Servers). This

should be left blank if a proxy is not used.

16 | P a g e

SocksHostname Address of the SOCKS proxy host (or IP address). Leave

blank if SOCKS is not used.

SocksPort Port of the SOCKS proxy. Leave blank if SOCKS is not used.

SocksUsername SOCKS proxy username. Leave blank if SOCKS is not used.

SocksPassword SOCKS proxy password. Leave blank if SOCKS is not used.

SocksVersion Set to “4” if using SOCKS4 proxy and “5” if using SOCKS5

proxy. Leave blank if SOCKS is not used.

GdmUserName User name for login to the GDM Server.

Note: By default, if blank, this uses the windows account

running the service (see Running as a service). If service is

running as ‘Network Service’ then a user with Genic-Server

authentication must be created in GDM for this e.g.

gdmxdist@domain.

GdmPassword Password for login to the GDM Server.

GdmUseKerberosAuthentication Use this if Kerberos authentication is to be used by GDM.

GdmUsername setting will be ignored if this setting is

“true” and the service will send a Kerberos corresponding

to the Log On user account of the service. Not applicable to

GDM before version 4.

InboxDirectory Location of Inbox directory e.g. ‘D:\GDMX\Inbox’ or

‘\\fileserver\Gdmx\Inbox’. This defaults to a directory called

“Inbox” located as a subdirectory of the parent of the

location where the application GdmxDistributionService.exe

resides.

LogsDirectory Location of logs directory e.g. ‘D:\GDMX\Logs’ or

‘\\fileserver\Gdmx\Logs’. This defaults to a directory called

“Logs” located as a subdirectory of the parent of the

location where the application GdmxDistributionService.exe

resides.

NOTE: This is obsolete as of version 1.3 of the service.

Logging has been improved with log4net logging libraries.

DisableArchiving “true” or “false”. Set to true to disable archiving of the

GDMX files. Gdmx files will be deleted after successful load

17 | P a g e

to GDM server.

DeliveryLogRecycleInterval This is the interval of deletion of the log file, which contains

names of files that have been already loaded. This is used in

conjunction with disabling archiving of GDMX. This is to

prevent duplicate GDMX files being reloaded. 30 days

should keep file reasonably small and allow more than

enough time to deal with GDMX loading errors.

Interval This is the number of minutes that the service should check

for new GDMX files in the GdmxDocumentPath.

MaxFileSizeKb This allows large GDMX files to be split into smaller files

before attempting delivery. This setting should not be

required. Set to 0 to avoid splitting the files. Otherwise files

will be split of greater than MaxFileSizeKb kilobytes.

MaxFileStorageDir When files are delivered they get archived in a storage

location in a directory called “Delivered” within the “Inbox”

directory. To avoid the directories from having too many

files, this is set to control that number. Default is 2000 if

this is set to 0 or non positive number.

NextFileWaitSeconds This is the number of seconds you want the application to

pause before delivering the next file. This was incorporated

to allow the GDM server a chance to garbage collect when

delivering a high volume of updates.

TimeoutSeconds This is the timeout setting for delivery to GDM server.

ClientName DataGenic will provide this and it will correlate with a

name used to identify you in our monitoring system.

Note: One of ReportByWebService or

DeleteSourceFileByWebService must be set to ‘true’.

EnableSourceFileDelete “true” or “false”. Set to true to delete the source file in

GdmxDocumentPath.

This value will be set to “true” when connecting to the

DataHub.

DeleteSourceFileByWebService “true” or “false”. Set to true if using a proxy server to access

18 | P a g e

GdmxDocumentPath.

EnableReporting “true” or “false”. Set to true to allow DataGenic to receive

log files indicating if data was successfully delivered.

ReportByWebService “true” or “false”. Set to true if using a proxy server to access

GdmxDocumentPath.

ReportLogPath FTP path to where to upload report files. Only required if

reporting by FTP.

WebProxyAddress Address of web proxy server if one is used to access the

DataGenic DataHub server for reporting data loads and

deleting data files that have been loaded.

WebProxyUserName Web proxy username – leave blank if web proxy not used.

WebProxyPassword Web proxy password – leave blank if web proxy not used.

New for version 1.7.1.0 The “collectionPaths” settings are an addition to the service in version 1.7.1.0. If configured correctly, these settings allow the service to collect GDMX from a backup FTP server, should the primary FTP server be out of service. This configuration looks like the following: <collectionPaths> <paths> <add name="Primary" path="ftp://ftp1.datagenic.net/outbound" userName="user1" password="password1" /> <add name="Secondary" path="ftp://ftp2.datagenic.net/outbound" userName="user2" password="password2" /> </paths> </collectionPaths> The “path” setting should be the ftp location of the GDMX files. The “user” and “password” settings are the user name and password for the FTP account allowing access to the “path” FTP address. These settings will be provided by Datagenic for services that are to collect GDMX files from the Datagenic Datahub. The “name” field is a name for the FTP but it is important to know that the path labelled as “Primary” is the main FTP server and the other path is the backup FTP server. The service will always try to get GDMX files from the primary FTP server first, and only if the service cannot connect to that FTP will it attempt to connect and collect GDMX files from the secondary FTP account. The tag below must exist under the “configSections” tag. <section name="collectionPaths"

19 | P a g e

type="GdmxDistributionService.Config.CollectionSection,GdmxDistributionService" /> It is tidy to set the “GdmxDocumentPath” setting under “appSettings” to “”, should there be paths configured under “collectionPaths” as the paths here will override any setting in the GdmxDocumentPath setting. This setting can still be used for file shares and local directories for chained Distribution Services which are collecting from a share that was populated by a main Distribution Service which got its files from the Datagenic DataHub.

20 | P a g e

7.2 DataSource-config.xml <?xml version="1.0" encoding="utf-8" ?>

<DataSources>

<DataSource enabled="true">

<Name>European Central Bank (ECB) - Foreign Exchange</Name>

<Prefix>ECB_FX</Prefix>

</DataSource>

<DataSource enabled="true">

<Name>European Energy Exchange (EEX) - Auction</Name>

<Prefix>EEXAuction</Prefix>

</DataSource>

<DataSource enabled="false">

<Name>S0142 GZip files</Name>

<Prefix>regex[S0142.*\.gz]</Prefix>

</DataSource>

<DataSource enabled="false">

<Name>European Energy Exchange (EEX) - Futures</Name>

<Prefix>EEXFutures</Prefix>

<TransformUri>transform://CUSTOM_PROPERTIES?CATEGORY=ECB_FXW</TransformUri>

</DataSource>

</DataSources>

Each ‘DataSource’ has an ‘enabled’ attribute for true/false value. Set to ‘false’ allows you to have the

data source listed in the file but disable the collection of that data source.

The ‘DataSource’ node contains a ‘Name’ and ‘Prefix’ setting. The ‘Name’ is simply a name to call the

data source, but the ‘Prefix’ must match the name of the GDMX file Prefix.

e.g. The prefix for ECB_FX_20081212100530.xml is ECB_FX.

From 1.7.0.0, the ‘Prefix’ can be configured to identify and process compressed files. In the 3rd data

source listed above you can see the <Prefix> tag listed with the following Regular Expression pattern:

regex[S0142.*\.gz]

This will match files with a S0142 prefix and a ‘.gz’ file extension: e.g . S0142_20121217-1000.gz

The last data source example (shown above) shows how we can import data with an XSL transform

action performed before the import action is executed. This is done by creating another call to the

21 | P a g e

GDM Server (WebService "transformAndImportFileContent"). To specify that this Web Service

method should be used instead of "importFileContent", a "TransformUri" should be added to the

"DataSource-config.xml" configuration, with the full uri of the transform required.

The “transform://” type, containing the XSL, must exist in GDM. Also, the files collected by the

GDMX Distribution

Service must contain valid GDMX content. The intended use of this functionality is to make custom

changes to the GDMX imported by converting the existing GDMX content collected into some

custom GDMX content

e.g. adding additional model properties.

22 | P a g e

7.3 Distribution-config.xml <DistributionConfiguration>

<Delivery enableReporting=”true”>

<name>Production</name>

<location>http://localhost:8080/</location>

<DataSources>

<Prefix>CFTC_FUT</Prefix>

<Prefix>CFTC_FUTOPT</Prefix>

<Prefix>ECB_FX</Prefix>

<Prefix>EEXAuction</Prefix>

<Prefix>EEXContinuous</Prefix>

<Prefix>EEXFutures</Prefix>

</DataSources>

</Delivery>

<Delivery>

<name>Test</name>

<location>\\testserver\incoming</location>

<DataSources>

<Prefix>CFTC_FUT</Prefix>

<Prefix>CFTC_FUTOPT</Prefix>

<Prefix>ECB_FX</Prefix>

<Prefix>regex[S0142.*\.gz]</Prefix>

<Prefix>regex[BMRA.*\.gz]</Prefix>

<Prefix>EEXFutures</Prefix>

</DataSources>

</Delivery>

</DistributionConfiguration>

Each Delivery node requires an ‘enableReporting’ attribute set to ‘true’ if upload of logs facility is to

be used for that delivery. This is usually to send a log to DataGenic indicating that the loading of the

GDMX succeeded or not and thus allowing DataGenic to monitor the collection of data.

The Delivery node has a name, location and DataSources nodes. Name is simply a name for the

23 | P a g e

delivery while ‘location’ is the URL or path to where the GDMX is to be delivered. If the location

starts with ‘http’ it is expected that the intended delivery is to a GDM server therefore the full URL

to the GDM web service is required. Otherwise it is expected to be a local directory or network share

(in the UNC format: \\server\share). If a network share, ensure that the account in which the service

is run has permission to write to the share.

The DataSources node contains a list of Prefix nodes, each of which, contain the GDMX prefix for a

GDMX file to be delivered.

From 1.7.0.0, as with the Data Source configuration in the previous section, you have the option of

configuring the prefix with a regular expression. This also allows you to match file extensions, such

as ‘gz’ for compressed files. The format is identical to the data source configuration:

<Prefix>regex[S0142.*\.gz]</Prefix>

24 | P a g e

8 Logging Configuration

Logging is configured using the file “config\log4net.config” (bin\

GdmxDistributionService.exe.log4net before version 1.7.4.1). The example below will output a log

file for each day with name of format “Logs<yyyyMMdd>.txt”.

<?xml version="1.0" encoding="utf-8" ?>

<log4net>

<appender name="RollingFile" type="log4net.Appender.RollingFileAppender">

<file value="Logs" />

<appendToFile value="true" />

<rollingStyle value="Date" />

<datePattern value="yyyyMMdd'.txt'" />

<staticLogFileName value="false"/>

<layout type="log4net.Layout.PatternLayout">

<conversionPattern value="%date %level %logger -

%message%newline%exception" />

</layout>

</appender>

<root>

<level value="INFO" />

<appender-ref ref="RollingFile" />

</root>

</log4net>

The level of logs to be written to log is at INFO level. The logging can be reconfigured without

restarting the GDMX Distribution Service, allowing the level to be switched to DEBUG level if needed

to diagnose a problem.

By default, a daily log file is created and written to in the “Logs” directory location which contains

step by step activity of what the service is doing. There is also a daily log in the “Logs\Deliveries”

location. This is an XML file containing info, such as delivered time and delivery name, about GDMX

files that were successfully delivered. In the Logs\ FailedDeliveries location there is a single XML file

containing information about failed deliveries.

25 | P a g e

9 Appendix A – File & Directory Listing

GdmxDistributionService\bin

ChilkatDotNet4.dll Required for SFTP communication if SFTP is used

instead of FTP.

DataGenic.DataManager.DataAccess.dll Required for sending data to GDM

DataGenic.DataManager.dll Required for sending data to GDM

DataSource-config.xml Data sources configuration file which determines

which datasources are enabled for collection by the

service. ( From 1.7.4.1, no longer used, see details

of config directory below )

Deliveries.xsl XSL Stylesheet used in transforming the Delivery log

files to HTML for viewing in browser.

Distribution-config.xml Recipient configuration for distribution of the GDMX

files. ( From 1.7.4.1, no longer used, see details of

config directory below )

FailedDeliveries.xsl XSL Stylesheet used in transforming the Failed

Delivery log file to HTML for viewing in browser.

GdmServerErrors.xml Error message configuration

GdmxDistributionService.exe The service executable.

GdmxDistributionService.exe.config The configuration file for the executable, which

contains general settings for the service. ( From

1.7.4.1, no longer used, see details of config

directory below )

GdmxDistributionService.exe.log4net Log4Net logging configuration file. ( From 1.7.4.1, no

longer used, see details of config directory below )

ICSharpCode.SharpZipLib.dll Compression libraries.

Log4net.dll Log4Net library.

Log4net.xml Required by the Log4Net library.

Microsoft.Web.Service3.dll Required for web service communication

GdmxDistributionService\config (From

1.7.4.1)

App.config The configuration file for the executable, which

contains general settings for the service.

26 | P a g e

DataSource-config.xml Data sources configuration file which determines

which datasources are enabled for collection by the

service.

Distribution-config.xml Recipient configuration for distribution of the GDMX

files.

log4net.config Log4Net logging configuration file.

GdmxDistributionService\install

install.bat Installs and starts the service

readme.txt Instruction on how to install the service and changes

made by release version.

uninstall.bat Stops and uninstalls the service.

27 | P a g e

10 Appendix B – Troubleshooting

Here are some quick answers to commonly asked questions with regards to validating that the

service is running and whether it has processed GDMX files.

For the purpose of this guide, the Inbox and Logs directories are by default sub directories of the

parent of the executable directory.

10.1 Have all downloaded files been delivered? Look in the Inbox directory. If there are any files there, then that GDMX has not been delivered to all

recipients.

10.2 Has ECB_FX_20080606120030.xml delivered? Look in the Inbox\Delivered directory. If an ECB_FX GDMX has been delivered then there will be a

subdirectory ECB_FX. Under that there is a sequence of sub directories starting at a to z. There is a

control mechanism to ensure that there are not too many files in each directory and when this

number is reached in one directory a new directory in the sequence is created. If this file has been

delivered it will exist in one of these sub directories, so search for files with

ECB_FX_20080606120030 in the name.

10.3 What GDMX files have been delivered on 12th November 2008?

Look in the Logs\Deliveries directory. Find file Deliveries_20081012.xml i.e. “Deliveries_” followed by

date in yyyyMMdd format and “.xml”. If the executable has not been moved since this file was

created and “Deliveries.xsl” exists in the executable directory, open the Deliveries_yyyymmdd.xml

file with Internet Explorer and it should display a table of deliveries for this date.

10.4 How to determine if there are failed deliveries?

If a GDMX file was successfully downloaded, and it failed to be delivered to all recipients, it should

still be in the Inbox directory. Look in the Logs\FailedDeliveries directory for a file called

FailedDeliveries.xml. A failed delivery to a recipient gets logged in this file. It gets logged three times

and stops logging, However the Service still attempts to deliver the GDMX file. The reason (error

message) for the failed delivery of the file is contained in the FailedDeliveries.xml file. If the

executable has not been moved since this file was created and “FailedDeliveries.xsl” exists in the

executable directory, open the FailedDeliveries.xml file with Internet Explorer and it should display a

table of failed deliveries.

28 | P a g e

10.5 How to find logs of activity of the service? Look in the Logs directory. Each log file contains a log for one day, hence the naming

“LogsyyyyMMdd.txt”. This is a text file and so can be opened in Notepad or WordPad. Since version

1.3 the location of the logs will depend on the configuration of log4net, which was introduced to

improve the logging options.

10.6 How to Start / Stop the GDMX Distribution Service

Go to Control Panel -> Administrative Tools -> Services.

There should be a service in the list called “DataGenic GDMX Distributor”. The first tab, General, has

buttons to start and stop the service.

You can also use ‘net stop “GDMX Distribution Service”’ or ‘sc \\myServer stop “GDMX Distribution

Service”’ via the command line.

NOTE: The GDMX service should not be stopped while in the process of delivering files.