Messages can be too large for the application, queue, or queue manager. Depending on the environment, WebSphere MQ provides a number of ways of dealing with messages that are longer than 4 MB.
On WebSphere MQ for z/OS, you can increase the MaxMsgLength attribute, up to 100 MB. This value should be set to reflect the size of the messages using the queue. On other MQSeries or WebSphere MQ Version 5 products you can:
Each of these approaches is described in the remainder of this section.
The MaxMsgLength queue manager attribute defines the maximum length of a message that can be handled by a queue manager. Similarly, the MaxMsgLength queue attribute is the maximum length of a message that can be handled by a queue. The default maximum message length supported depends on the environment in which you are working.
If you are handling large messages, you can alter these attributes independently. The attribute value can be set between 32768 bytes and 100 MB, or between 0 bytes and 100 MB on WebSphere MQ for z/OS.
After changing one or both of the MaxMsgLength attributes, restart your applications and channels to ensure that the changes take effect.
Note that on WebSphere MQ for z/OS you are permitted to change the queue MaxMsgLength attribute, but not the queue manager MaxMsgLength attribute (which is fixed at 100 MB).
When these changes are made, the message length must be less than or equal to both the queue and the queue manager MaxMsgLength attributes. However, existing messages may be longer than either attribute.
If the message is too big for the queue, MQRC_MSG_TOO_BIG_FOR_Q is returned. Similarly, if the message is too big for the queue manager, MQRC_MSG_TOO_BIG_FOR_Q_MGR is returned.
This method of handling large messages is easy and convenient. However, consider the following factors before using it:
Not supported in WebSphere MQ for z/OS.
Increasing the maximum message length as discussed in topic "Increasing the maximum message length" has some negative implications. Also, it could still result in the message being too large for the queue or queue manager. In these cases, a message can be segmented. For information about segments, see Message groups.
The next sections look at common uses for segmenting messages. For putting and destructively getting, it is assumed that the MQPUT or MQGET calls always operate within a unit of work. It is strongly recommended that this technique is always used, to reduce the possibility of incomplete groups being present in the network. Single-phase commit by the queue manager is assumed, but of course other coordination techniques are equally valid.
Also, in the getting applications, it is assumed that if multiple servers are processing the same queue, each server executes similar code, so that one server never fails to find a message or segment that it expects to be there (because it had specified MQGMO_ALL_MSGS_AVAILABLE or MQGMO_ALL_SEGMENTS_AVAILABLE earlier).
This is the simplest scenario, in which one application puts a message to be retrieved by another. The message may be large: not too large for either the putting or the getting application to handle in a single buffer, but possibly too large for the queue manager or a queue on which the message is to be put.
The only changes necessary for these applications are for the putting application to authorize the queue manager to perform segmentation if necessary,
PMO.Options = (existing options) MQPUT MD.MsgFlags = MQMF_SEGMENTATION_ALLOWED
and for the getting application to ask the queue manager to reassemble the message if it has been segmented:
GMO.Options = MQGMO_COMPLETE_MSG | (existing options) MQGET
The application buffer must be large enough to contain the reassembled message (unless the MQGMO_ACCEPT_TRUNCATED_MSG option is included).
If data conversion is necessary, it may have to be done by the getting application specifying MQGMO_CONVERT. This should be straightforward because the data conversion exit is presented with the complete message. Attempting to do data conversion in a sender channel will not be successful if the message is segmented, and the format of the data is such that the data-conversion exit cannot carry out the conversion on incomplete data.
Application segmentation is used for two main reasons:
However, if data conversion is not an issue, or if the getting application always uses MQGMO_COMPLETE_MSG, queue-manager segmentation can also be allowed by specifying MQMF_SEGMENTATION_ALLOWED. In our example, the application segments the message into four segments:
PMO.Options = MQPMO_LOGICAL_ORDER | MQPMO_SYNCPOINT MQPUT MD.MsgFlags = MQMF_SEGMENT MQPUT MD.MsgFlags = MQMF_SEGMENT MQPUT MD.MsgFlags = MQMF_SEGMENT MQPUT MD.MsgFlags = MQMF_LAST_SEGMENT MQCMIT
If you do not use MQPMO_LOGICAL_ORDER, the application must set the Offset and the length of each segment. In this case, logical state is not maintained automatically.
The getting application cannot, or chooses not to, guarantee to have a buffer that will hold any reassembled message. It must therefore be prepared to process segments individually.
For messages that are segmented, this application does not want to start processing one segment until all of the segments that constitute the logical message are present. MQGMO_ALL_SEGMENTS_AVAILABLE is therefore specified for the first segment. If you specify MQGMO_LOGICAL_ORDER and there is a current logical message, MQGMO_ALL_SEGMENTS_AVAILABLE is ignored.
Once the first segment of a logical message has been retrieved, MQGMO_LOGICAL_ORDER is used to ensure that the remaining segments of the logical message are retrieved in order.
No consideration is given to messages within different groups. If such messages do occur, they are processed in the order in which the first segment of each message appears on the queue.
GMO.Options = MQGMO_SYNCPOINT | MQGMO_LOGICAL_ORDER
| MQGMO_ALL_SEGMENTS_AVAILABLE | MQGMO_WAIT
do while ( SegmentStatus == MQSS_SEGMENT )
MQGET
/* Process each remaining segment of the logical message */
...
MQCMIT
The messages must be maintained in logical order in a group, and some or all of them may be so large that they require application segmentation.
In our example, a group of four logical messages is to be put. All but the third message are large, and require segmentation which is performed by the putting application:
PMO.Options = MQPMO_LOGICAL_ORDER | MQPMO_SYNCPOINT MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP | MQMF_SEGMENT MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP | MQMF_SEGMENT MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP | MQMF_LAST_SEGMENT MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP | MQMF_SEGMENT MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP | MQMF_LAST_SEGMENT MQPUT MD.MsgFlags = MQMF_MSG_IN_GROUP MQPUT MD.MsgFlags = MQMF_LAST_MSG_IN_GROUP | MQMF_SEGMENT MQPUT MD.MsgFlags = MQMF_LAST_MSG_IN_GROUP | MQMF_SEGMENT MQPUT MD.MsgFlags = MQMF_LAST_MSG_IN_GROUP | MQMF_LAST_SEGMENT MQCMIT
In the getting application, MQGMO_ALL_MSGS_AVAILABLE is specified on the first MQGET. This means that no messages or segments of a group are retrieved until the entire group is available. When the first physical message of a group has been retrieved, MQGMO_LOGICAL_ORDER is used to ensure that the segments and messages of the group are retrieved in order:
GMO.Options = MQGMO_SYNCPOINT | MQGMO_LOGICAL_ORDER
| MQGMO_ALL_MESSAGES_AVAILABLE | MQGMO_WAIT
do while ( (GroupStatus != MQGS_LAST_MSG_IN_GROUP) ||
(SegmentStatus != MQGS_LAST_SEGMENT) )
MQGET
/* Process a segment or complete logical message. Use the GroupStatus
and SegmentStatus information to see what has been returned */
...
MQCMIT
You can put and get a segmented message that spans a unit of work in a similar way to Putting and getting a group that spans units of work.
You cannot, however, put or get segmented messages in a global unit of work.
Not supported in WebSphere MQ for z/OS.
This method allows a large object to be transferred from one node to another without the need for the object to be stored on WebSphere MQ queues at either the source or the destination nodes. This is of particular benefit where the data already exists in another form, for example, for mail applications.
To do this, you need to specify a message exit at both ends of a channel. For information on how to do this, see WebSphere MQ Intercommunication.
WebSphere MQ defines the format of a reference message header (MQRMH). See WebSphere MQ Application Programming Reference for a description of this. This is recognized by means of a defined format name and may or may not be followed by actual data.
To initiate transfer of a large object, an application can put a message consisting of a reference message header with no data following it. As this message leaves the node, the message exit retrieves the object in an appropriate way and appends it to the reference message. It then returns the message (now larger than before) to the sending Message Channel Agent for transmission to the receiving MCA.
Another message exit is configured at the receiving MCA. When this message exit sees one of these messages, it creates the object using the object data that was appended and passes on the reference message without it. The reference message can now be received by an application and this application knows that the object (or at least the portion of it represented by this reference message) has been created at this node.
The maximum amount of object data that a sending message exit can append to the reference message is limited by the negotiated maximum message length for the channel. The exit can only return a single message to the MCA for each message that it is passed, so the putting application can put several messages to cause one object to be transferred. Each message must identify the logical length and offset of the object that is to be appended to it. However, in cases where it is not possible to know the total size of the object or the maximum size allowed by the channel, the sending message exit can be designed so that the putting application just puts a single message, and the exit itself puts the next message on the transmission queue when it has appended as much data as it can to the message it has been passed.
Before using this method of dealing with large messages, consider the following:
See WebSphere MQ Application Programming Reference for a description of the fields in the reference message header and the message descriptor.
In the MQMD structure, the Format field must be set to MQFMT_REF_MSG_HEADER. The MQHREF format, when requested on MQGET, is converted automatically by WebSphere MQ along with any bulk data that follows.
Here is an example of the use of the DataLogicalOffset and DataLogicalLength fields of the MQRMH:
A putting application might put a reference message with:
Assuming that the object is 70,000 bytes long, the sending message exit sends the first 40,000 bytes along the channel in a reference message containing:
It then places another message on the transmission queue containing:
When this message exit is seen by the sending message exit, the remaining 30,000 bytes of data is appended, and the fields are set to:
The MQRMHF_LAST flag is also set.
For a description of the sample programs provided for the use of reference messages, see Chapter 35, Sample programs (all platforms except z/OS).