R-GMA

Producers and Archiver

Paul Mealor

edg-pinger-timeping

edg-netmon-rgma-info

Producer API

edg-gridftplog2rgma

Producer APILog Here be servlets

Overview

AC DBP

P

P

P

edg-netmon2rgma

Producer API

edg-netmon-archiver

Archiver API

MySQLDB

gsiwuftp

FIFO

edg-netmon2rgma

• Maintains producers for the tools– Tools’ scripts are rerun for each measurement batch– Program running APIs must be persistent

• Files– /opt/edg/sbin/edg-netmon2rgma.pl– /opt/edg/var/edg-netmon2rgma/edg-netmon2rgma-fifo– /etc/rc.d/init.d/edg-netmon2rgmad

• Configuration– RGMA client configuration

Tool/netmon2rgma interaction

•addmetric pinger rtt 10 NMIdSource=pc18.hep.ucl.ac.uk tool=pinger

PingER Normallog

edg-netmon2rgma

•CircularBufferProducer_new(“NetworkRTT”,“WHERE NMIdSource=‘pc18.hep.ucl.ac.uk’ AND tool=‘pinger’”);

•CircularBufferProducer_setRemoteBufferSize($producer,10*2);

Web interface

If producer doesn’t exists

If bufferSize < count * 4 / 3

Tool/netmon2rgma interaction

•measurement pinger rtt pc18.hep.ucl.ac.uk edgnm.gridpp.rl.ac.uk pinger 100 57

PingER Normallog

edg-netmon2rgma

•CircularBufferProducer_insert($producer,”INSERT INTO NetworkRTT(NMIdSource, NMIdDestination, tool, packetSize, value)VALUES(pc18.hep.ucl.ac.uk, edgnm.gridpp.rl.ac.uk, pinger, 100, 57)”);

Web interface

edg-netmon2rgma

• Possibly should rewrite in Java– Problem in C (Perl) API results in memory

leak and heavy performance hit– Current solution is to restart the daemon with

a cron job• /etc/cron.d/edg-netmon2rgma-restart

– But java lacks the text-handling of Perl– Would have to have java on the NM

Tools modifications

• Generally 2 new functions• initialise_rgma

– Writes initialisation directives to the FIFO– i.e. fixed columns, metric names, expected number of

entries• Netmon2rgma creates producers, changes buffer sizes when

necessary

• produce_it– Writes the results of the measurements

• No extra configuration– Assumes /opt/edg/var/edg-netmon2rgma/edg-

netmon2rgma-fifo

edg-netmon-rgma-info

• Publishes NetworkSE, NetworkCE tables– Republishes every 12 hours (i.e. keepalive)

• Files– /opt/edg/sbin/edg-netmon-rgma-info.pl– /opt/edg/etc/edg-netmon-rgma-info.conf

• Local NM, associated SEs and CEs

– /etc/rc.d/init.d/edg-netmon-rgma-infod

• RGMA client configuration

edg-ftlog2rgma

• Publishes file-transfer logs (i.e. GridFTP)

• Files– /opt/edg/sbin/edg-gridftplog2rgma.pl– /etc/rc.d/init.d/edg-gridftplog2rgmad

• Reads from GridFTP log– /var/log/gsiwuftpd.log– Publishes new entries when they are added

• RGMA client configuration

edg-netmon-archiver

• Program– /opt/edg/share/edg-netmon-archiver.jar– /opt/edg/sbin/edg-netmon-archiver

• runs the Java code

– /opt/edg/bin/edg-netmon-archiver-setup• Configures MySQL database; requires mysql root password• What to do in LCFG?

– /opt/edg/etc/edg-netmon-archiver.conf• List of tables to archiver plus predicates for each

– /etc/rc.d/init.d/edg-netmon-archiverd– Also RGMA client configuration

edg-netmon-archiver

• Ironically this might be the best package to write in C or Perl– Archiver API is all but unaffected by C API problem– But might run on R-GMA servlet machine

• Interaction with edg-netmon-info– Archives every row that edg-netmon-info produces– Perhaps need to automatically remove duplicates (via

mysql)

Package progress

• Tools, edg-netmon2rgma, edg-netmon-rgma-info– 99%– Latest tweaks still need to be uploaded

• edg-ftlog2rgma– 75%– rc script, autobuild process

• edg-netmon-archiver– 95%– need to upload, tag, check build process (ant)

Testing

• Tools on pc18.hep.ucl.ac.uk• edg-netmon2rgmad on pc18.hep.ucl.ac.uk• “User servlets” on pc23.hep.ucl.ac.uk• Registry and Schema servlets on

pc23.hep.ucl.ac.uk– Something always goes wrong when I try to connect

to the WP3 Registry at infocat.gridpp.rl.ac.uk• >1200 RTT/loss measurements archived by

Tuesday• >130 TCP throughput measurements to

localhost :)

GridFTP logging in MDS

Paul Mealor

Status

• Information provider– Will be in EDG 2 on SE

• Publishes the most recent information for each host pair

• MDS being phased out in EDG 2– GIIS structure replaced by R-GMA– Will be all R-GMA later

Details

• Publishes achieved throughput– Filesize, time etc as parameters

• Publishes most recent transfer between each host pair

• Specifies source and host pair, plus, if known, source and host NM– NM names taken from config file as with other netmon

info providers

• Uses same schema as other netmon– But doesn’t use the full tree structure

Schemadn: NMMeasureId=$source/$dest,$opt{rootdn}objectClass: NetworkMeasurementNMMeasureId: $source/$destSourceHost: $sourceSourceNE: $sourceNEDestHost: $destDestNE: $destNEMetricName: gridftpthroughputMetricValue: $throughputMetricUnit: bit/sMonitorTool: GridFTPMetricTime: $timestampParameter: time: $etimeParameter: size: $sizeParameter: buffersize: $bufferParameter: streams: $streamsParameter: stripes: $stripescreateTimestamp: $timestampmodifyTimestamp: $timestamp

END