User ManualCG Elastic

Version 1.17-SNAPSHOT

2017-11-30

Table of Contents1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1

2. Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  3

2.1. HeartBeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  3

2.2. MetricBeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  4

2.3. CgWmBeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  6

2.3.1. ISHome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  6

2.3.2. ISService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  8

2.3.3. UmQueue. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  9

2.3.4. UmDoc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  10

2.3.5. Tablespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  12

2.3.6. Perflog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  13

2.3.7. Sysmem, Syscpu, Sysdisk (deprecated) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  15

2.4. Jenkins. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  16

Table 1. History

Date Author Detail

2017-10-09 bcouetil Minor changes

2017-10-06 bcouetil CgElastic beat migration + CgUtils DB update

2017-09-15 bcouetil Anomaly code refactoring

1. IntroductionwebMethods offer some nice KPI and dashboard handling by the hand of Optimize For Process,Optimize For Infra, and MashZone.

The setup, from which CgElastic is a part of, involves the (free and versatile) Elastic Stack instead(Elasticsearch, Logstash, Beats, Kibana).

Figure 1. webMethods Architecture involving Elastic Stack

1

Figure 2. Kibana dashboard on a webMethods project

Figure 3. Graphana dashboards on a webMethods project

2

2. ModulesIn the following paragraphs, different modules are presented. All have the purpose to populate anElastisearch. Some are official modules applied to webMethods ecosystem.

2.1. HeartBeatThe official Elastic module HeartBeat is able to ping a host regularly through various channels. Itcan be used to check that the IS, UM, MWS, CentraSite modules are up and running, as wells ascontinuous deployment tools such as Gerrit, Jenkins, SonarQube, Nexus.

Figure 4. Number of healthchecks KO checked every 30s

Example JSON Data

{  "_index": "heartbeat-2017.09.28",  "_type": "doc",  "_id": "AV7IZ1_jh9aXLVWW6fIY",  "_version": 1,  "_score": null,  "_source": {  "@timestamp": "2017-09-28T12:12:50.727Z",  "beat": {  "hostname": "2008c14c7d19",  "name": "2008c14c7d19",  "version": "5.5.2"  },  "duration": {  "us": 22723  },  "host": "frpardge",  "response": {  "status": 200  },  "type": "Kibana",  "up": true,  "url": "http://frpardge:5601/app/kibana#/management?_g=%28%29"  }}

3

2.2. MetricBeatThe official Elastic module MetricBeat gives insights on Memory, CPU, disk, network of the currenthost. It is able to give these insight on Docker server and containers as well.

Figure 5. CPU usage overtime

Figure 6. Memory usage overtime

Figure 7. Docker CPU usage overtime

4

Figure 8. Docker memory usage overtime

Example JSON Data

{  "_index": "metricbeat-2017.09.28",  "_type": "doc",  "_id": "AV7IZkNlh9aXLVWW6e3m",  "_version": 1,  "_score": null,  "_source": {  "@timestamp": "2017-09-28T12:11:37.813Z",  "beat": {  "hostname": "frpardge",  "name": "frpardge",  "version": "5.5.2"  },  "docker": {  "info": {  "containers": {  "paused": 0,  "running": 6,  "stopped": 3,  "total": 9  },  "id": "KRQF:O4Q2:C3XT:HVLP:LKJW:RO5A:XB55:BKTR:K5RB:LK5R:63EJ:6BPB",  "images": 82  }  },  "metricset": {  "host": "/var/run/docker.sock",  "module": "docker",  "name": "info",  "rtt": 39687  },  "type": "metricsets"  }}

5

2.3. CgWmBeatCapgemini webMethods beat module cgwmbeat gathers all custom statistics from webMethodsservers. There are several area of data detailed in next paragraphs.

2.3.1. ISHome

The ISHome module gives insights on Integration Server overall statistics

Figure 9. IS Memory usage

Figure 10. IS uptime

6

Example JSON Data

{  "_index": "isstat",  "_type": "isstat",  "_id": "AV7IaOwuh9aXLVWW6fiJ",  "_version": 1,  "_score": null,  "_source": {  "startTime": "2017-09-28 12:36:40 CEST",  "uptime": "1h:37m:52s",  "maxMem": 932352,  "totalMem": 631808,  "freeMem": 364499,  "usedMem": 267309,  "freeMemPer": 57,  "usedMemPer": 43,  "memArrayMax": 801792,  "svrT": 2,  "svrTMax": 14,  "srvThreadCount": 1,  "srvThreadCountPeak": 11,  "sysT": 374,  "sysTMax": 375,  "conn": 55,  "connMax": 55,  "connAvg": 322,  "reqTotal": 31,  "reqLifetime": 3714,  "reqAvg": 165,  "reqTotAvg": 209,  "newReqPM": 36,  "endReqPM": 36,  "errSvc": 104,  "errSys": 43,  "svcRate": 100,  "ssnUsed": 3,  "ssnPeak": 4,  "ssnMax": 2147483647,  "connStatefulSessionsCurrent": 0,  "connStatefulSessionsPeak": 0,  "connStatefulSessionsLimit": 2147483647,  "insertDate": "2017-09-28T12:14:33.005"  },  "fields": {  "insertDate": [  1506600873005  ],  "usedMemPretty": [  267309000  ],  "totalMemPretty": [  631808000  ],  "maxMemPretty": [  932352000  ]  },  "sort": [  1506600873005  ]}

7

2.3.2. ISService

The ISService module gathers webMethods services running statistics.

Figure 11. IS Services total access

Example JSON Data

{  "_index": "servicestat",  "_type": "startup",  "_id": "AV7TWV-fh9aXLVWWGoGl",  "_version": 1,  "_score": null,  "_source": {  "fullName": "wm.xslt.Admin:startup",  "package": "wm.xslt.Admin",  "shortName": "startup",  "accessLastDate": "2017-09-28 20:31:46 CEST",  "accessTotal": 1,  "accessNew": 0,  "accessRunning": 0,  "insertDate": "2017-09-30T15:13:23.359"  },  "fields": {  "insertDate": [  1506784403359  ]  },  "sort": [  1506784403359  ]}

8

2.3.3. UmQueue

The UmQueue module gathers Universal Messaging channels and topics statistics. This helps know,on publish/subscribe processes, if queues are growing, or if the platform handles everything at thehighest possible speed.

Figure 12. UM queues length

Example JSON Data

{  "_index": "umqueue",  "_type": "StartRouteMsg",  "_id": "AV7TXczxh9aXLVWWGpYr",  "_version": 1,  "_score": null,  "_source": {  "name": "StartRouteMsg",  "type": "channel",  "currentNoOfConnections": 1,  "currentNumberOfEvents": 0,  "totalConsumed": 12,  "totalPublished": 4,  "usedSpace": 781,  "memoryUsage": 0,  "consumedRate": 0,  "fanoutTime": 0,  "insertDate": "2017-09-30T15:18:13.489"  },  "fields": {  "insertDate": [  1506784693489  ]  },  "sort": [  1506784693489  ]}

9

2.3.4. UmDoc

The UmDoc module is the main module for business dashboards. It tracks every single (business)document published in the system.

Figure 13. CG Anomalies by type

Figure 14. Document published by type

10

Example JSON Data

{  "_index": "pubdoc",  "_type": "DfArle",  "_id": "AV7TTTGmh9aXLVWWGkmc",  "_version": 1,  "_score": null,  "_source": {  "DGE.Entete.Nature.Type": "DiffusionFichier",  "DGE.Entete.Nature.Version": "02.01",  "DGE.Entete.Nature.Phase": "DF_ARLE",  "DGE.Entete.Service.Reference": "M0002E001",  "DGE.Entete.Service.Version": "02.01",  "DGE.Entete.Echange.Identifiant": "CNAM.DGEARLE.Q1785945",  "DGE.Entete.Echange.Horodatage": "2017-09-30T17:00:04.911",  "DGE.Entete.Echange.RefConversation": "DGEA011701",  "DGE.Entete.Echange.Fonction": "9",  "DGE.Entete.Emetteur.Identifiant": "99640000",  "DGE.Entete.Emetteur.Type": "RFO",  "DGE.Entete.Emetteur.Libelle": "DGE",  "DGE.Entete.Recepteur.Identifiant": "01110021",  "DGE.Entete.Recepteur.Type": "RFO",  "DGE.Entete.Recepteur.Libelle": "CNAM",  "DGE.Document.EchangeOrigine.Identifiant": "RGF011701",  "DGE.Document.EchangeOrigine.Horodatage": "2017-09-30T17:00:00.203",  "DGE.Document.Resultat.Statut": "R",  "DGE.Document.Resultat.Complement[0].Code": "9302",  "DGE.Document.Resultat.Complement[0].Libelle": "Adhesion producteur inconnue",  "insertDate": "2017-09-30T15:00:05.157"  },  "fields": {  "DGE.Document.EchangeOrigine.Horodatage": [  1506790800203  ],  "ModuleCode": [  "null/null"  ],  "DGE.Entete.Echange.Horodatage": [  1506790804911  ],  "insertDate": [  1506783605157  ]  },  "sort": [  1506783605157  ]}

11

2.3.5. Tablespace

The Tablespace module gives insights on Oracle TBS Occupation

Figure 15. Tablespaces occupation overtime

Figure 16. Tablespaces occupation gauges

Example JSON Data

{  "_index": "dbstat",  "_type": "WEBMDATA",  "_id": "AV7TUF8eh9aXLVWWGlge",  "_version": 1,  "_score": null,  "_source": {  "schema": "WEBMDATA",  "allocatedMo": 200,  "freeMo": 179.81,  "usedMo": 20.19,  "freePercent": 89.91,  "usedPercent": 10.09,  "insertDate": "2017-09-30T15:03:33.406"  },  "fields": {  "insertDate": [  1506783813406  ],  "usedPercentPretty": [  0.10090000152587891  ]  },  "sort": [  1506783813406  ]}

12

2.3.6. Perflog

The perflog module listen to the cg-utils file that stores top level java services duration. With thisdata, you can see what services are taking too long to execute, and if the current duration isconsistent with the average.

Figure 17. Java service duration overtime

Figure 18. Java services duration heatmap

13

Example JSON Data

{  "_index": "logstash-2017.09.30",  "_type": "perflog",  "_id": "AV7TTUT9h9aXLVWWGknX",  "_version": 1,  "_score": null,  "_source": {  "offset": 76128,  "method": "sendDfArle",  "input_type": "log",  "source": "/islogs/log4j2/perfs.log",  "message": "2017-09-30 17:00:07.451 INFO PERFORMANCES - #sendDfArle(...): in 473.41ms",  "type": "log",  "tags": [  "beats_input_codec_plain_applied"  ],  "duration": 473.41,  "@timestamp": "2017-09-30T15:00:07.451Z",  "@version": "1",  "beat": {  "hostname": "c843aa83715b",  "name": "c843aa83715b",  "version": "5.5.2"  },  "host": "c843aa83715b",  "timestamp": "2017-09-30 17:00:07.451"  },  "fields": {  "@timestamp": [  1506783607451  ]  },  "sort": [  1506783607451  ]}

14

2.3.7. Sysmem, Syscpu, Sysdisk (deprecated)

sysmem, syscpu and sysdisk provides insight of memory, CPU and disk usage from the serverhosting the current IS server.

They are deprecated since the unveil of the Elastic module metricbeat which is more feature-rich.

15

2.4. JenkinsThe Jenkins plugin Logstash gives insight on build launches and durations.

Figure 19. Jenkins builds duration

Example JSON Data

{  "_index": "jenkins",  "_type": "build",  "_id": "AV7PQahNh9aXLVWWB8FS",  "_version": 1,  "_score": null,  "_source": {  "data": {  "id": "753",  "result": "FAILURE",  "projectName": "CNAV-DGE_P1_Review",  "displayName": "#753",  }  },  "message": [  "[...truncated 18.72 KB...]",  "[withMaven] Jenkins Task Scanner Plugin not found, don't display results of source code scanning for 'TODO' and'FIXME' in pipeline screen.",  "[Pipeline] // withMaven",  "[Pipeline] }",  "[Pipeline] // stage",  "[Pipeline] emailextrecipients",  "[Pipeline] step",  "[Pipeline] }",  "[Pipeline] // timeout"  ],  "source": "jenkins",  "source_host": "https://bpmfactory.s2-eu.capgemini.com/jenkins/",  "@buildTimestamp": "2017-09-29T22:07:32+0200",  "@timestamp": "2017-09-29T22:09:00+0200",  "@version": 1  }}

16