Messaging Services Update

Lionel Cons IT/SDC

20 May 2014

Messaging Services @ CERN

§  1st era: “by the Grid for the Grid” §  only EGEE (then EGI) services §  primitive management and monitoring

§  2nd era: “service diversification” §  non-Grid services added (managed by IT/PES) §  improved management and monitoring

§  3rd era: “service consolidation” §  no more EGI services §  integrated management and monitoring

20  May  2014  Messaging  Services  Update   2  

Messaging Services @ CERN

§  1st era: “by the Grid for the Grid” §  ActiveMQ + Java6 + SL5 + physical machines §  Quattor + Nagios + YAIM

§  2nd era: “service diversification” §  ActiveMQ + Apollo + RabbitMQ §  Java6 + Java7 (+ Erlang) §  SL5 + SL6 §  physical machines + virtual machines §  Quattor + Nagios + MBSD (and friends)

§  3rd era: “service consolidation” §  ActiveMQ + Java7 + SL6 + physical machines §  Puppet + Agile Monitoring + MBSD (and friends)

20  May  2014  Messaging  Services  Update   3  

ActiveMQ vs. Apollo

§  Apollo is ActiveMQ rewritten from scratch, by the ActiveMQ experts themselves

§  Very fruitful collaboration in the OpenLab spirit: “you make it, we break it”

§  Red Hat buying ActiveMQ added delays §  Some Apollo features have been backported:

§  LevelDB message store

§  runtime configuration changes

20  May  2014  Messaging  Services  Update   4  

Physical vs. Virtual Machines

§  Brokers are heavy disk I/O users §  to comply to JMS, persistent messages must

be “persisted to disk” (i.e. sync’ed to disk)

§  Most of the problems having affected the messaging services have been caused by the use of virtual machines

§  What do we win by using virtual machines?

20  May  2014  Messaging  Services  Update   5  

Java  Applica*on  

Java  Virtual  Machine  

Opera*ng  System  

Java Application Stack

20  May  2014  Messaging  Services  Update   6  

CPU   RAM   disk   NIC  

ActiveMQ on a Virtual Machine

20  May  2014  Messaging  Services  Update   7  

NAS  CPU   RAM   disk   NIC  

Opera*ng  System  

Virtual  Machine  Applica*on   Other  Apps  

OS  

JVM  

Ac*veMQ  

Other  VMs  Apps  

Multiple ActiveMQ on a Physical Machine

20  May  2014  Messaging  Services  Update   8  

Ac*veMQ  

JVM  

Opera*ng  System  

CPU   RAM   disk   NIC  

Other  Apps  JVM   JVM  

Ac*veMQ   Ac*veMQ  

Management Consolidation

§  Most of the software we need comes from SL + EPEL §  we publish 14 packages in EPEL

§  All the rest is built using CERN’s Koji §  All our machines will be Puppet-managed

§  this is work-in-progress

§  We are using MBSD which can be seen as LANDB at the messaging level

20  May  2014  Messaging  Services  Update   9  

Monitoring Consolidation

§  We will stop using Nagios §  We will integrate as much as possible with

the “Agile Monitoring” offering §  we however have specific needs such as

aggregation-based alerts or per-client alerting

§  The final solution will be fully integrated with GNI and Roger

§  We are investigating how much can be done by Esper

20  May  2014  Messaging  Services  Update   10  

Messaging Services Consolidation

§  Joint effort from IT/PES and IT/SDC §  Good progress so far

§  Should be finished by the end of the year §  new services to be deployed by end of June §  monitoring migration will take longer

§  existing services to be migrated too §  ActiveMQ 5.5 to 5.9 (with Red Hat’s additions)!

20  May  2014  Messaging  Services  Update   11  

Next Steps

§  Once the new tools are in place, we can better define the operational procedures

§  Then will come SLDs and SLAs §  Then we will chase the messaging brokers

not run by us…

20  May  2014  Messaging  Services  Update   12