Mysql active mq_failover

MYSQL ActiveMQ FailOver Setup

This document explains the master slave setup of active mq with MYSQL as shared database. At any given time, only one broker in such setup will be holding lock on the database and thus will behave as a master. All other brokers are slaves and pause waiting for the exclusive lock. For the applications depending on active mq, such setup will help have a seamless switch over mechanism without impacting any active business processes.

1 ActiveMQ Setup

A master slave setup can have one master and multiple slave nodes as shown below.

However, this document shall explain the scenario of setting up one master and one slave.

We use jdbc persistence to make MYSQL as shared database among master,slave.

There are two ways to active this. Database Locker Lease Database

Let us see, how they differ:

1.1 Database Locker

The Database Locker is the default locker for the JDBC persistence adapter. It locks a database table in a transaction to ensure that only single resource is used. You can configure it like this:<persistenceAdapter>

<jdbcPersistenceAdapter dataDirectory="${activemq.data}" dataSource="#mysql-ds" lockKeepAlivePeriod="10000">

<locker>

<database-locker lockAcquireSleepInterval="5000"/>

</locker>

</jdbcPersistenceAdapter>

</persistenceAdapter>

The Database Locker uses its keepAlive method to ensure the broker still holds the lock. You can set the keep alive period using thelockKeepAlivePeriod property. The default period is 30000 ms. If a broker fails to acquire the lock on the database, it will retry every lockAcquireSleepInterval milliseconds.

This locker opens a JDBC transaction against a database table (activemq_lock) that lasts as long as the broker remains alive. This locks the entire table and prevents another broker from accessing the store. In most cases this will be a fairly long running JDBC transaction which occupies resources on the database over time.

A problem with this locker can occur when the master broker crashes or loses its connection to the database. The information about the lock remains in the database, until the database responds to the half closed socket connection via a tcp timeout. The database lock expiry requirement can prevent the slave from starting for a period. In addition, if the database supports failover, and the connection is dropped in the event of a replica failover, that JDBC transaction will be rolled back. The broker sees this as a failure and both master and slave will again compete for a lock.

1.2 Lease Database Locker

The Lease Database Locker solves the master/slave problems of the default Database Locker. It does not open a long running JDBC transaction but lets the master broker acquire a lock that's only valid for a short period. To retain the lock the master broker must periodically extend the lock's lease. The slave broker also checks periodically to see if the lease has expired. The lease can survive a db replica failover. You can configure it like this:<persistenceAdapter>


<locker>

<lease-database-locker lockAcquireSleepInterval="10000"/>

</locker>



In order for this mechanism to work correctly, each broker in the master/slave pair must have a different brokerName attribute defined on thebroker tag or use the lease-database-locker leaseHolderId attribute, as it is this value that is used to reserve a lease.

The lease based lock is acquired by blocking at startup. It is then retained for a period whose duration (in ms) is given by thelockKeepAlivePeriod attribute. To retain the lock the master broker periodically extends its lease by lockAcquireSleepInterval milliseconds each time. In theory, therefore, the master broker is always (lockAcquireSleepInterval - lockKeepAlivePeriod) ahead of the slave broker with regard to the lease. It is imperative that lockAcquireSleepInterval > lockKeepAlivePeriod, to ensure the lease is always current. As of ActiveMQ 5.9.0 a warning message is logged if this condition is not met.

In the simplest case, the clocks between master and slave must be in sync for this solution to work properly. If the clocks cannot be in sync, the locker can use the system time from the database CURRENT TIME and adjust the timeouts in accordance with their local variance from the db system time. If maxAllowableDiffFromDBTime is greater than zero the local periods will be adjusted by any delta that exceedsmaxAllowableDiffFromDBTime.Icon

It is important to know if the default rules your JDBC driver uses for converting TIME values are JDBC compliant. If you're using MySQL, for example, the driver's JDBC URL should contain useJDBCCompliantTimezoneShift=true to ensure that TIME value conversion is JDBC compliant. If not the locker could report a large time difference when it compares the retrieved lease expiration time against the current system time. Consult your JDBC driver's manual for more details.

2 Active MQ Configuration

Start by setting up two instances of active mq following the installation guide.

For simplicity, we refer these two instances as broker-1, broker-2 throughout the document.

In each of the broker, edit the following in conf/activemq.xml:

http://activemq.apache.org/installation.html

2.1 Add JdbcPersistentAdapter:

<persistenceAdapter>


<locker><lease-database-locker

lockAcquireSleepInterval="10000"/></locker>



2.2 Add TransportConnector:

<transportConnectors>  <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600" </transportConnectors>

Define different port numbers for each of the brokers. For eg, broker-1’s port will be 61616 and broker-2’s port be 61626.

2.3 Datasource configuration:

Define the datasource. We shall use the MYSQL database:



<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">

<property name="driverClassName" value="com.mysql.jdbc.Driver"/>

<property name="url" value="jdbc:mysql://127.0.0.1:3306/activemq?relaxAutoCommit=true"/>

<property name="username" value="activemquser"/><property name="password" value=" activemquser "/><property name="poolPreparedStatements" value="true"/>

</bean>

2.4 Configure Jetty :Edit the conf/jetty.xml in order to have two different jetty ports for broker-1,broker-2.For e.g, you may use 8161 for broker-1 and 8162 for broker-2.

<bean id="jettyPort" class="org.apache.activemq.web.WebConsolePort" init-method="start">  <property name="host" value="0.0.0.0"/> <property name="port" value="8161"/> </bean>

3 Application configuration

In order to have applications connecting to active mq do a seamless switchover to the healthy broker i.e master, use failover transport as shown below:

failover:(tcp://broker1:61616, tcp://broker2:61626 )

Note: The switch over time will be in close to maximum of lockAcquireSleepInterval of the jdbc adapter configuration time for a slave to become master. Till then you active mq connector should be waiting for a master node to come up.

tcp://broker2:61616

Sample configuration for Mule’s ActiveMQ connector:

4 Testing Master/Slave

Bring up broker1 first. As seen below, broker-1 will try to acquire the lock on the database and thus will become the master.

Now bring up broker2. It notices that database lock is held by broker-1 and cannot become the master. Thus, will keep retrying to get the lock, per the interval configured.

Now, if you bring down the master, slave will acquire the lock and become master. It is to be noted that broker-2 will now act as master, despite broker-1 is up and running. Broker-1 will continue to poll for the lock till either broker-2 goes down or loose the db lock.

Your activemq connector should now be able to seamlessly switch over to available master node and start sending messages to it. Below is a snapshot of durable messages being stored on DB.

Mysql active mq_failover

Software

Transcript of Mysql active mq_failover