HACMP

Top

url MC91 - high availability for MQ on Unix. Install into /MQHA/bin : samples use it.

This SupportPac provides notes and sample scripts to assist with the installation and configuration of WebSphere MQ (WMQ) V6 and V7 in High Availability (HA) environments. Three different platforms and environments are described here, but they share a common design and this design can also be extended for many other systems.

Specifically this SupportPac deals with the following HA products:

• HACMP (High Availability Cluster Multi Processing)
• Veritas Cluster Server (VCS)
• MC/ServiceGuard (MCSG)

MC91 installation :

MC91 configuration :

• [1] Configure the HA Cluster
• [2] Configure the shared disks
• [3] Create the Queue Manager
• [4] Configure the movable resources
• [5] Configure the Application Server or Agent
• [6] Configure a monitor

1. Configure TCP/IP on the cluster nodes for HACMP. Remember to configure ~root/.rhosts, /etc/rc.net, etc.
2. Configure the cluster, cluster nodes and adapters to HACMP as usual.
3. Synchronise the Cluster Topology.

This step creates the volume group (or disk group) and filesystems needed for the queue manager. So that this queue manager can be moved from one node to another without disrupting any other queue managers, you should designate a group containing shared disks which is used exclusively by this queue manager and no others. For performance, it is recommended that a queue manager uses separate filesystems for logs and data. The suggested layout therefore creates two filesystems within the volume group.

You can optionally protect each of the filesystems from disk failures by using mirroring or RAID.

Mount points must all be owned by the mqm user.

You will need the following filesystems:

• Per node:
  • /var on a local non-shared disk - this is a standard filesystem or directory which will already exist.
• Per queue manager:
  • /MQHA/<qmgr>/data on shared disks - this is where the queue manager data directory will reside.
  • /MQHA/<qmgr>/log on shared disks - this is where the queue manager recovery logs will reside.

The steps are :

1. Create the volume group that will be used for this queue manager's data and log files.
2. Create the /MQHA/<qmgr>/data and /MQHA/<qmgr>/log filesystems using the volume group created above.
3. For each node in turn, import the volume group, vary it on, ensure that the filesystems can be mounted, unmount the filesystems and varyoff the volume group.

1. Select a node on which to perform the following actions
2. Ensure the queue manager's filesystems are mounted on the selected node.
3. Create the queue manager on this node, using the hacrtmqm script
4. Start the queue manager manually, using the strmqm command
5. Create any queues and channels
6. Test the queue manager
7. End the queue manager manually, using endmqm
8. On the other nodes, which may takeover the queue manager, run the halinkmqm script

The resource group will use the IP address as the service label. This is the address which clients and channels will use to connect to the queue manager.

1. Create a resource group and select the type as discussed above.
2. Configure the resource group in the usual way adding the service IP label, volume group and filesystem resources to the resource group.
3. Synchronise the cluster resources.
4. Start HACMP on each cluster node in turn and ensure that the cluster stabilizes, that the respective volume groups are varied on by each node and that the filesystems are mounted correctly.

The queue manager is represented within the resource group by an application server or agent.

1. Define an application server which will start and stop the queue manager. The start and stop scripts contained in the SupportPac may be used unmodified, or may be used as a basis from which you can develop customized scripts. The examples are called hamqm_start and hamqm_stop.
2. Add the application server to the resource group definition created in the previous step.
3. Optionally, create a user exit in /MQHA/bin/rc.local
4. Synchronise the cluster configuration.
5. Test that the node can start and stop the queue manager, by bringing the resource group online and offline.

You can configure an application monitor which will monitor the health of the queue manager and trigger recovery actions as a result of MQ failures, not just node or network failures. Recovery actions include the ability to perform local restarts of the queue manager or to cause a failover of the resource group to another node.

To benefit from queue manager monitoring you must define an Application Monitor. If you created the queue manager using hacrtmqm, then one of these will have been created for you, in the /MQHA/bin directory, and is called hamqm_applmon.$qmgr.

1. To enable queue manager monitoring, define a custom application monitor for the Application Server created in previous step, providing the name of the monitor script and tell HACMP how frequently to invoke it. Set the stabilisation interval to 10 seconds, unless your queue manager is expected to take a long time to restart. This would normally be if your environment has long-running transactions that might cause a substantial amount of recovery/replay to be required.
2. To configure for local restarts, specify the Restart Count and Restart Interval.
3. Synchronise the cluster resources.
4. Test the operation of the application monitoring, and in particular verify that the local restart capability is working as configured. A convenient way to provoke queue manager failures is to identify the Execution Controller process (called amqzxma0) associated with the queue manager, and kill it.

Conclusion : the files we have to copy into /hacmp/, and adapt for our system, are :

• hamqm_start
• hamqm_stop
• hamqm_applmon.$qmgr

Then, using smitty, we have to

• define an application server
• define a custom application monitor

Top

url IC91 - high availability for MB on distributed platforms. Install into /MQHA/bin : samples use it.

A broker runs as a pair of processes, called bipservice and bipbroker. The latter in turn creates the execution groups that run message flows. It is this collection of processes which are managed by HA Software.

When creating the queue manager, don't configure the application server or application monitor described in SupportPac MC91. You will create an application server that covers the broker, queue manager and broker database instance.

When creating channels between queue managers, the sender channel should use the service address of the broker resource group and the broker queue manager's port number.

• General Configuration
  • [0] Configure the cluster
  • Configure the shared disks
• Configuration steps for Broker - UNIX
  • [1] Create and configure the queue manager. Create the resource group.
  • [2] Create and configure the broker database
  • [3] Create the message broker
  • [4] Place the broker under cluster control

1. Configure TCP/IP on the cluster nodes as described in your cluster software documentation.
2. Configure the cluster, cluster nodes and adapters to HA Software as usual.
3. Synchronise the Cluster Topology.
4. Now would be a good time to create and configure the user accounts that will be used to run the database instances, brokers and UNS. Home directories, (numeric) user ids, passwords, profiles and group memberships should be the same on all cluster nodes.

1. On one node, create a clustered queue manager as described in SupportPac MC91, using the hacrtmqm command. Use the volume group that you created for the broker and place the volume group and queue manager into a resource group to which the broker will be added. Don't configure the application server or application monitor described in SupportPac MC91 - you will create an application server that covers the broker, queue manager and broker database instance.
2. Set up queues and channels between the broker queue manager and the Configuration Manager queue manager:
   • On the Configuration Manager queue manager create a transmission queue for communication to the broker queue manager. Ensure that the queue is given the same name and case as the broker queue manager. The transmission queue should be set to trigger the sender channel.
   • On the Configuration Manager queue manager create a sender and receiver channel for communication with the broker queue manager. The sender channel should use the service address of the broker resource group and the broker queue manager's port number.
   • On the broker queue manager create a transmission queue for communication to the Configuration Manager queue manager. Ensure that the queue is given the same name and case as the Configuration Manager queue manager. The transmission queue should be set to trigger the sender channel.
   • On the broker queue manager create sender and receiver channels to match those just created on the Configuration Manager queue manager. The sender channel should use the IP address of the machine where the Configuration Manager queue manager runs, and the corresponding listener port number.
3. If you are using a UNS, set up queues and channels between the broker queue manager and the UNS queue manager:
   • On the broker queue manager create a transmission queue for communication to the UNS queue manager. Ensure that the queue is given the same name and case as the UNS queue manager. The transmission queue should be set to trigger the sender channel.
   • On the broker queue manager create a sender and receiver channel for communication with the UNS queue manager. If the UNS is clustered, the sender channel should use the service address of the UNS resource group and the UNS queue manager's port number.
   • On the UNS queue manager create a transmission queue for communication to the broker queue manager. Ensure that the queue is given the same name and case as the broker queue manager. The transmission queue should be set to trigger the sender channel.
   • On the UNS queue manager create a sender and receiver channel for communication with the broker queue manager, with the same names as the receiver and sender channel just created on the broker queue manager. The sender channel should use the service address of the broker resource group and the broker queue manager's port number.
4. Test that the above queue managers can communicate regardless of which node owns the resource groups they belong to.

There are two options regarding where the broker database is run, either inside or outside the cluster. If you choose to run the database outside the cluster then simply follow the instructions in the WMB documentation for creating the broker database but ensure that you consider whether the database is a single point of failure and make appropriate provision for the availability of the database.

1. Create the broker on the node hosting the logical host using the hamqsicreatebroker command.
2. Ensure that you can start and stop the broker manually using the mqsistart and mqsistop cmmands.
3. On any other nodes in the resource group's nodelist (i.e. excluding the one on which you just created the broker), run the hamqsiaddbrokerstandby command to create the information needed by these nodes to enable them to host the broker.

1. Create an application server which will run the broker, its queue manager and the database instance, using the example scripts provided in this SupportPac. The example scripts are called hamqsi_start_broker_as and hamqsi_stop_broker_as.
2. You can also specify an application monitor using the hamqsi_applmon.<broker> script created by hamqsicreatebroker. An application monitor script cannot be passed parameters, so just specify the name of the monitor script. Also configure the other application monitor parameters, including the monitoring interval and the restart parameters you require.
3. Synchronise the cluster resources.
4. Ensure that the broker, queue manager and database instance are stopped, and start the application server.
5. Check that the components started and test that the resource group can be moved from one node to the other and that they run correctly on each node.
6. Ensure that stopping the application server stops the components.
7. With the application server started, verify that the HACMP local restart capability is working as configured. A convenient way to cause failures is to identify the bipservice for the broker and kill it.

HisCock HA MQ whitepaper

A key problem in using MQ clustering for high availability is the problem of stuck messages (in Xmit queue).

Message expiry. If the message isn't delivered to it's target by the time the end-user would have timed out, get it to self destruct.

If you've got an HA set up there are 2 main options:

1. Use HA software so "the queue manager" is presented on a single IP/port no matter where it happens to be running so the value in the TAB file is always true.
2. Use the TAB file to define multiple instances of "THEQM" which identify QMA, QMB, etc

First squema is like this one :

We have an active machine, AIX-1, running QM1 and MB1, and a passive machine, AIX-2, which is almost always stopped.

So, in order to improve this second machine utilization, we can create a second queue manager and a second broker on AIX-2, and place its backup replicas in AIX-1 :

Finaly, we join QM1 and QM2 in a MQ cluster, so while one machine in moving to its backup image, the source messages are still processed.

A n+1 arquitecture is also possible : to have "n" machines running, and 1 more being the backup of all those "n" machines - we guess they will fail one at a time !

Se instala MC91 (HACMP para MQ) y luego IC91 (HACMP para MB).

Top

To stop a queue manager running under MQ for UNIX© systems :

• Find the process IDs of the queue manager programs that are still running using the ps command.
  For example, if the queue manager is called QMNAME, use the following command:
  ps -ef | grep QMNAME
• End any queue manager processes that are still running. Use the kill command, specifying the process IDs discovered using the ps command.
• End the processes in the following order :

Note: processes that fail to stop can be ended using kill -9. If you stop the queue manager manually, FFSTs might be taken, and FDC files placed in /var/mqm/errors. Do not regard this as a defect in the queue manager.

The queue manager should restart normally, even after you have stopped it using this method.

URL (sys admin guide)

/MQHA/bin/hamqproc contains the list of processes to be killed by hamqm_stop_su :

Top

• Separate disks for data files and logs
  • While not essential for HA reasons, it is recommended for performance
  • All our examples use this configuration
• Channel state is hardened
  • Sender channels will be automatically restarted (if triggered)
  • Use the virtual IP address (or name) in the CONNAME
  • Requesters will not be auto-restarted
    • But a 'server' machine is less likely to have requester channels
• May need to use custom scripts or services to restart things like
  • Trigger monitors
  • Command server
  • Applications
• It's exactly the same as a normal queue manager restart
  • So non-persistent messages will disappear
• Long-running transactions may cause long-running restart
• if cluster is used, place FR's on HACMP machines

TMM04 {BCN}

Top

It can be done (newer stop_su) providing the names in a file : see here.

Top

The core is :

Just does a "ping qmgrname" :

Verifies few processes are still running :

Gracias, Vicente !

An easy way to monitor cluster events and messages is by tailing the following HACMP log files:

• /tmp/hacmp.out - Outputs detailed logs on cluster events and output from application start and stop scripts. Note that output from the monitoring is not part of this log file.
• /usr/es/adm/cluster.log - Contains timestamped, formatted messages generated by HACMP start and stop scripts and daemons. This file contains only high level events, and not detailed log information. The above log files are local to each node.

Application monitoring in HACMP has its own set of log files.

Manual system start up tests :

1. Node 1 and node 2 are both active:
   1. disable the cluster on both nodes by using smitty clsstop
   2. stopping the cluster unmounts the shared drives. Mount the shared drives.
   3. start QM1 on node 1
   4. start QM2 on node 2
   5. observe the results to verify that no errors are reported during MQ operation.
2. Node 2 is the only active node:
   1. start QM1 on node 2
   2. start QM2 on node 2
   3. observe the results to verify that no errors are reported during MQ operation.
3. Node 1 is the only active node:
   1. start QM2 on node 1
   2. start QM1 on node 1
   3. observe the results to verify that no errors are reported during MQ operation.

The tests cases for verifying the HA configuration are:

1. Shared files - the following files should be located in shared directories:
   • MQ logs
   • Queue manager data for every queue manger should reside in a shared location.

/MQHA/<qmgr>/data and /MQHA/<qmgr>/log

Separate disks for data files and logs - while not essential for HA reasons, it is recommended for performance

The objective of this test suite is to verify if HACMP is able to start, restart, and monitor all the individual applications that are part of the cluster.

1. Automatic system startup/restart under HACMP control
   • end message broker gracefully : "mqsistop -i MB_NAME"
   • end message broker abruptly
     ps -ef | grep bipservice | grep -v grep | \ awk '{print $2}' | xargs kill -9
   • end queue manager gracefully : "endmqm -i QM_NAME"
   • end queue manager abruptly
     ps -ef | grep AMQXSSVN.EXE | grep -v grep | \ awk '{print $2}' | xargs kill -9
   • end queue manager abruptly : kill "AMQZXMA0", Execution Controller process. url
   • disable qmgr restarting by:
     • changing the owner of AMQERR01.LOG
     • chmod 400 of active log file S0000000.LOG
     • renaming general mqs.ini or specific qm.ini

2. Restart attempts setting

   Verify the number of retry attempts. Each WebSphere Business Integration application will be restarted three times before a resource group failover is initiated. The number of retry attempts can be configured in HACMP.

3. Failover

4. Fallback

   Fallback refers to the movement of a resource group from a secondary or a failover node to the primary node, which is being reintegrated into the cluster. In the current WebSphere Business Integration cluster, automatic fallback is disabled. However, manual reintegration should be validated:

   1. Bring node 1 down:
      shutdown -r now
   2. Node 1 resource group fails over to node 2.
   3. Verify failover by looking at the HACMP log file:
      tail -f /tmp/hacmp.out
   4. Start up cluster on node 1 after node 1 is back up:
      smitty clstart
   5. Observe that the resource groups are still running on node 2 even though the cluster on node 1 is backed up.
   6. Repeat the above test for node 2 fallback.

Assuming a two-node active/active cluster, the steps are

1. Select one machine to upgrade first
2. At a suitable time, when the moving of a queue manager will not cause a serious disruption to service, manually force a migration of the active queue manager to its partner node
3. On the machine that is now running both queue managers, disable the failover capabilities for the queue managers.
4. Upgrade the software on the machine that is not running any queue managers
5. Re-enable failover, and move both queue managers across to the newly upgraded machine
6. Disable failover again
7. Upgrade the original box
8. Re-enable failover
9. When it will cause least disruption, move one of the queue managers across to balance the workload

Top

• Fix Packs : move both environments to a single machine and apply maintenance to "empty" machine. Failback all components and apply maintenance to the other machine. Caution : there is a point of "no way back" !
• design applications to handle duplicates
  if store & forward plus persistency is used, when we introduce disaster recovery, a duplicate message chance is created. if EndToEnd transaction number is used (and in E2E_Ack also), the duplicate problem is solved and persistency is not required anymore.
• design applications to be able to run in multiple instances
• a PR (partial repository) is a subscriber of the FR's (full repositories) and (if there are 2) it receives duplicates
• how to monitor HACMP is working ok ?
  Include checking all of the things that the qmgr must have to satisfy the business applications, like: the status of channels, listeners, availability of application queues (put/get enabled, trigger attributes), existence of processes, namelists, trigger monitor, applications, ...
• maybe you dont trust any "process monitoring" and prefer to put a message into a queue to validate the qmgr is working !
• script to go to backup node :
  • endmqm (in background)
  • wait some time
  • kill remaining mq resources in specific order (see SysAdmin manual)
• CM does not require to go under HACMP : not many changes to production environment. CM can go on a VMWARE image !

Top

HACMP-MB es IC91 : High Availability for WebSphere Message Broker on Distributed Platforms

Highly Available WebSphere Business Integration Solutions redbok.

All Support Packs

HACMP on AIX : complete list (***), migration checklist

HACMP install, etc

Impact 2008 HA.

HA & disaster recovery chat