Distributed (JMS) messaging applications
using GlassFish and MantaRay
Distributed (JMS) messaging applications
using GlassFish and MantaRay
Generic JMS RA (1.7) that is bundled with GlassFish V2 enables
applications deployed on GlassFish to use almost any JMS 1.1 compliant
message broker. This article talks about how applications (typically
Message Driven Beans) deployed in a GlassFish cluster can use
MantaRay peer to peer architecture for
their JMS messaging needs using
Generic resource adapter for JMS.
MantaRay is an open source enterprise-grade messaging middleware
solution that is based on a peer-to-peer architecture. With its
peer-to-peer topology, MantaRay eliminates the need for costly central
messaging brokers.
Figure 1: GlassFish
and MantaRay
The steps/configuration mentioned below are based on the following
versions :
Project GlassFish V2 Build 30
https://glassfish.dev.java.net/downloads/v2-b30.html
MantaRay Java 2.0.1
Generic JMS RA 1.7
(Bundled with GlassFish, under
GLASSFISH_HOME/lib/addons/resourceadapters/genericjmsra/genericra.rar)
OR
https://genericjmsra.dev.java.net/files/documents/3308/43334/genericra.rar
Initial
Setup
Refer to MantaRay documentation for installing MantaRay.
Typically it involves extracting the zip file into a folder, which
would be denoted as MANTA_HOME in this document.
No changes are required to
the configuration files for a basic setup.
For the purpose of this
document MantaRay is installed under MANTA_HOME, GlassFish V2 is
installed under GLASSFISH_HOME.
Install GlassFish application server and create a cluster, the
domain that is used for this cluster creation should have been created
using the cluster profile, the
following steps briefly describe the tasks for 3 instance cluster
creation.
# NODE-AGENT CREATION
GLASSFISH_HOME/bin/asadmin
create-node-agent --user <adminUser> --passwordfile
<passwordFileLocation> --savemasterpassword=true agent1
#START NODE-AGENT
GLASSFISH_HOME/bin/asadmin
start-node-agent --user <adminUser> --passwordfile
<passwordFileLocation> agent1
# BEGIN CLUSTER CREATION
GLASSFISH_HOME/bin/asadmin
create-cluster --port <domainAdminPort> --user <adminUser>
--passwordfile <passwordFileLocation> --autohadb=false
cluster1
# CREATE INSTANCE1
GLASSFISH_HOME/bin/asadmin
create-instance --port <domainAdminPort> --user <adminUser>
--passwordfile <passwordFileLocation> --nodeagent ra-agent
--cluster cluster1
--systemproperties HTTP_LISTENER_PORT=1111:IIOP_LISTENER_PORT=33700
instance1
# CREATE INSTANCE2
GLASSFISH_HOME/bin/asadmin
create-instance --user admin --port <domainAdminPort>
--passwordfile <passwordFileLocation> --nodeagent agent1
--cluster cluster1
--systemproperties HTTP_LISTENER_PORT=1112:IIOP_LISTENER_PORT=33701
instance2
# CREATE INSTANCE3
GLASSFISH_HOME/bin/asadmin
create-instance --user admin --port <domainAdminPort>
--passwordfile <passwordFileLocation> --nodeagent agent1
--cluster cluster1
--systemproperties HTTP_LISTENER_PORT=1113:IIOP_LISTENER_PORT=33702
instance3
- Modify the cluster1's classpath to add MantaRay jars
located in MANTA_HOME folder. The asadmin GUI could
be used to
modify a domain's classpath. Open a browser and type the url of the
application server admin GUI - https://hostname:adminport. Go to
Configuration -> cluster1-config -> JVM Settings -> Path
Settings . Add
an
entry for the jar files shown below in
the classpath suffix.
MANTA_HOME/ext/antlr.jar
MANTA_HOME/manta.jar
MANTA_HOME/ext/clink-1.5.0.jar
MANTA_HOME/ext/commons-logging.jar
MANTA_HOME/ext/log4j-1.2.8.jar
- Modify the cluster1's JVM option to add MantaRay configuration
file as a JVM propert. The asadmin GUI could
be used to do this. Open a browser and type the url of the
application server admin GUI - https://hostname:adminport. Go to
Configuration -> cluster1-config -> JVM Settings -> JVM
options .
-DmantaConfig=MANTA_HOME/config/default_config.xml
- Edit the server.policy file in the
[GLASSFISH_HOME/domains/<domain>/config/] directory using your
favourite text editor and add the following line to the default grant
block.
permission java.util.logging.LoggingPermission "control";
permission
java.util.PropertyPermission "*",
"read,write";
Should you use an application client in your application,
edit
the application client's client.policy file in the
[GLASSFISH_HOME/lib/appclient/client.policy] directory and
add the following line to it.
permission javax.security.auth.PrivateCredentialPermission
"javax.resource.spi.security.PasswordCredential * \"*\"","read";
- Create a File system JNDI object store to bind ManataRay JMS
administered objects. The following code snippet creates a FS
object store and binds the required MantaRay objects to the jndi tree.
import javax.naming.*;
import java.util.Properties;
public class Main {
public Main() {
}
public static void main(String[] args) {
try {
Properties props = new Properties();
props.put("java.naming.factory.initial",
"com.sun.jndi.fscontext.RefFSContextFactory");
props.put("java.naming.provider.url",
"file:/<objectStoreFolderName>");
InitialContext jndiContext = new InitialContext(props);
MantaQueue mq = new MantaQueue("SampleQueue1");
MantaQueue outmq = new
MantaQueue("SampleQueue2");
MantaTopic top = new MantaTopic("SampleTopic");
MantaXAConnectionFactory confac = new
MantaXAConnectionFactory();
jndiContext.rebind("SampleQueue1", mq);
jndiContext.rebind("SampleQueue2",
outmq);
jndiContext.rebind("SampleTopic",
top);
jndiContext.rebind("mantaxaconnectionfactory", confac);
} catch (Exception e) {
System.out.println("Could not create JNDI " + "context: " +
e.toString());
System.exit(1);
}
}
}
Configuring GlassFish
cluster and MantaRay
Figure 2 : Configuration
The above configuration describes a simple MDB
application that is configured to listen to a MantaRay topic
SampleTopic, and when it receives a message from the topic it
sends the message to a queue SampleQueue2.
MantaRay is a messaging middleware solution
based on peer-peer architecture. MantaRay's support for JMS1.1 API has
been exploited to achieve the above configuration. Applications
deployed on GlassFish clusters (or instances) can use MantaRay's
peer-peer messaging capabilities through standard JMS apis using the
Generic JMS Resource Adapter. It supports both publish/subscribe and
point-point messaging models. Please refer to MantaRay documentation
for more details.
JNDI object store : MantaRay
administered objects are bound to a file system object store and
generic JMS ra is configured to use this object store. Other JNDI
stores (LDAP..) can also be used but their configuration is beyond the
scope of this document. A simple standalone program (mentioned in the
Initial Setup section) can be used to bind MantaRay administered
objects to the JNDI tree.
MantaRay configuration file :
The default configuration file -MANTA_HOME/config/default_config.xml is
used by all the MantaRay peers, since the default MantaRay
configuration file is designed to be used by multiple peers there are
no inherent port conflicts. The MantaRay peers would use ports in the
range of 6600 to 6700 (default_config.xml). If you feel the need for
configuring specific ports , then a separate configuration file has to
be created for every peer. A typical use case for this would arise when
you want to use the MantaRay management console, each peer would then
require a unique RMI port.
GlassFish Configuration :A
GlassFish cluster is created with 3 (or N ) instances. The cluster
configuration is modified to include MantaRay client jars in the
classpath and also a JVM property is added for mantaConfig file. When
an application (MDB) is deployed on the cluster it is deployed on all
instances that belong to the cluster and also presents a homogeneous
view to the user. Every instance of the application in the
cluster creates one MantaRay peer to consume or produce messages. The
MDB uses the standard JMS 1.1 APIs to send/receive messages. It
also uses generic JMS resource adapter which provides connection
pooling for the application.
Note: Restart the cluster after completing all the changes.
# Stopping the node agent stops all the instances in the cluster also.
GLASSFISH_HOME/bin/asadmin
stop-node-agent --user <adminUser> --passwordfile
<passwordFileLocation> agent1
# Start the node agent with sync instances set to true, this would
start the cluster with the updated configuration.
GLASSFISH_HOME/bin/asadmin
start-node-agent --user <adminUser> --passwordfile
<passwordFileLocation> --syncinstances=true agent1
Configuring
the Resource Adapter
- Add ${appserver-install-dir}/bin to your PATH. The asadmin
CLI
command can be found at ${appserver-install-dir}/bin. In GlassFish a
resource adapter configuration is used to specify the configuration of
a resource adapter. Use the following command to create a resource
adapter configuration for genericra, to configure it to work with
MantaRay Java 2.0.1.
asadmin
create-resource-adapter-config --user <adminname>
--password
<admin password> --property
SupportsXA=true:ProviderIntegrationMode=jndi:RMPolicy=OnePerPhysicalConnection:
JndiProperties=java.naming.factory.initial\=com.sun.jndi.fscontext.RefFSContextFactory
java.naming.provider.url\=file://space/mantarayobjects:LogLevel=FINEST
genericra
A brief description of the
various properties used in the above command is explained below:
SupportsXA
Set the supports distributed transactions attribute to true. The level
of transactional support the adapter provides -- none, local, or XA --
depends on the capabilities of the Enterprise Information System [EIS]
being adapted. If an adapter supports XA transactions and this
attribute is XA, the application can use distributed transactions to
coordinate the EIS resource with JDBC and JMS resources.
ProviderIntegrationMode
Set the integration mode as JNDI. Two integration modes exist in the
Generic Resource Adapter for JMS. The JNDI mode allows the resource
adapter to use the administered objects published in the message
provider's JNDI provider to integrate with the message provider.
JndiProperties
This property [comma-separated list of name-value pairs] specifies JNDI
provider properties to be used for
connecting to the JMS provider's JNDI. In our case, we set it to the
JNDI configuration specified earlier.
RMPolicy
Some XAResource implementations such as IBM MQ Series, relies
on a
Resource Manager per Physical Connection and this causes issues when
there is inbound
and
outbound communication to the same queue manager in a single
transaction (For example, an MDB sends a response to a
destination). When RMPolicy is set to OnePerPhysicalConnection, the
XAResource
wrapper implementation's isSameRM in Generic JMS RA would check if both
the XAResources use the same physical connection, before delegating to
the wrapped objects. Ensure that this attribute is set to
"OnePerPhysicalConnection" if the application uses XA.
Deploying
the Resource adapter
- Download the Generic RA bits from the project
site. With Glassfish V2, Generic
RA is available out-of-the-box with the application server and you
could choose to use the bundle resource adapter as well in the step
below.
- Deploy the resource adapter using the asadmin deploy
command, as shown below. In the image above, see Generic JMS
RA deployed in the application server.
$ asadmin deploy --user admin --password
adminadmin
<location of the generic resource adapter rar file>
Creating
Connection Factories and Administered Objects in GlassFish.
In order to configure a JMS Connection Factory, using the Generic
Resource Adapter for JMS, a Connector connection pool and resource
needs to be created in the application server, as shown below. In the
image above, see inpool [pointing to Generic JMS RA and QCF]
and jms/XAConFac [for inpool] created in the application server.
Connector
connection pool creation
#Creates a Connection Pool called inpool and points to
mantaxaconnectionfactory
asadmin create-connector-connection-pool -- raname genericra
connectiondefinition javax.jms.QueueConnectionFactory
--transactionsupport XATransaction --property
ConnectionFactoryJndiName=mantaxaconnectionfactory inpool
Connector
resource creation
#Creates a connector resource named jms/XAConFac and
binds this resource to JNDI for applications to use.
asadmin create-connector-resource --poolname inpool jms/XAConFac
Admin
Objects
For JMS Destination Resources, an administered object needs to be
created. In the image above, see jms/SampleQueue2 [pointing to Generic
JMS
RA and SampleQueue2] created in the application server.
#Creates a javax.jms.Queue Administered Object and binds it to
application server's JNDI tree at
jms/SampleQueue2 and points to SampleQueue2
asadmin create-admin-object --raname genericra --restype
javax.jms.Queue --property DestinationJndiName=SampleQueue2
jms/SampleQueue2
Component
Deployment descriptors
The deployment descriptors need to take into account the resource
adapter and the connection resources that have been created. A sample
sun-ejb-jar.xml for a Message Driven Bean that listens to a destination
called SampleTopic in MantaRay, and publishes back reply messages
to a
destination resource named jms/SampleQueue2 is shown below.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE sun-ejb-jar PUBLIC "-//Sun Microsystems, Inc.//DTD
Application Server 8.1 EJB 2.1//EN"
"http://www.sun.com/software/appserver/dtds/sun-ejb-jar_2_1-1.dtd">
<sun-ejb-jar>
<enterprise-beans>
<ejb>
<ejb-name>MyBean</ejb-name>
<jndi-name>sunMDB</jndi-name>
<resource-ref>
<res-ref-name>jms/MyQueueConnectionFactory</res-ref-name>
<jndi-name>
jms/XAConFac</jndi-name>
</resource-ref>
<resource-env-ref>
<resource-env-ref-name>jms/OutQueue</resource-env-ref-name>
<jndi-name>
jms/SampleQueue2</jndi-name>
</resource-env-ref>
<bean-pool>
<steady-pool-size>10</steady-pool-size>
<resize-quantity>2</resize-quantity>
<max-pool-size>30</max-pool-size>
<pool-idle-timeout-in-seconds>60</pool-idle-timeout-in-seconds>
</bean-pool>
<mdb-resource-adapter>
<resource-adapter-mid>genericra</resource-adapter-mid>
<activation-config>
<activation-config-property>
<activation-config-property-name>DestinationType</activation-config-property-name>
<activation-config-property-value>javax.jms.Topic</activation-config-property-value>
</activation-config-property>
<activation-config-property>
<activation-config-property-name>MaxPoolSize</activation-config-property-name>
<activation-config-property-value>30</activation-config-property-value>
</activation-config-property>
<activation-config-property>
<activation-config-property-name>RedeliveryAttempts</activation-config-property-name>
<activation-config-property-value>3</activation-config-property-value>
</activation-config-property>
<activation-config-property>
<activation-config-property-name>RedeliveryInterval</activation-config-property-name>
<activation-config-property-value>1</activation-config-property-value>
</activation-config-property>
<activation-config-property>
<activation-config-property-name>ReconnectAttempts</activation-config-property-name>
<activation-config-property-value>1000</activation-config-property-value>
</activation-config-property>
<activation-config-property>
<activation-config-property-name>ReconnectInterval</activation-config-property-name>
<activation-config-property-value>1</activation-config-property-value>
</activation-config-property>
<activation-config-property>
<activation-config-property-name>DestinationJndiName</activation-config-property-name>
<activation-config-property-value>SampleTopic
</activation-config-property-value>
</activation-config-property>
<activation-config-property>
<activation-config-property-name>ConnectionFactoryJndiName</activation-config-property-name>
<activation-config-property-value>mantaxaconnectionfactory</activation-config-property-value>
</activation-config-property>
</activation-config>
</mdb-resource-adapter>
</ejb>
</enterprise-beans>
</sun-ejb-jar>
The business logic encoded in Message Driven Bean could then lookup the
configured QueueConnectionFactory/Destination resource
to create a connection as shown below (in the onMessage(...) method).
Context context = null;
ConnectionFactory connectionFactory = null;
logger.info("In PublisherBean.ejbCreate()");
try {
context = new InitialContext();
queue = (javax.jms.Queue) context.lookup
("java:comp/env/jms/SampleQueue2");
connectionFactory = (ConnectionFactory)
context.lookup("java:comp/env/jms/MyQueueConnectionFactory");
connection =
connectionFactory.createConnection();
QueueSession qss =
connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
QueueSender sender = qss.createSender(queue);
TextMessage msg = qss.createTextMessage();
msg.setText(txtmsg.getText());
sender.send(msg);
connection.close();
} catch (Throwable t) {
logger.severe("PublisherBean.ejbCreate:"
+ "Exception: " +
t.toString());
}
Client
The setup can be tested using a simple client sample that comes along
with MantaRay
We will use the MANTA_HOME/samples/src/sample/jms/topics/ReliableChat
and MANTA_HOME/samples/src/sample/jms/queues/Talk as producer and
consumer application respectively
Complie the samples by executing the Compile.sh script that is provided
under the above folders, this should generate the classes for these
sample applications.
To see it in action, modify the
MANTA_HOME/samples/src/sample/jms/topics/ReliableChat/RunSample1.sh and
ensure that we publish to the correct topic, "-t SampleTopic".
Modify the MANTA_HOME/samples/src/sample/jms/queues/Talk/RunSample1.sh
and ensure that we receive from the correct queue, "-qr SampleQueue2"
Output of
MANTA_HOME/samples/src/sample/jms/topics/ReliableChat/RunSample1.sh
> ./RunSample1.sh
*** MantaRay Layer
Name: m10.12.171.1226603 ***
MantaRay 2.0.1
initialization completed.
Enter text messages
to clients that subscribe to the SampleTopic topic.
Press Enter to
publish each message.
Typing 'exit' will
stop the program.
Hi, this is a test
message
TOPICSENDER: Hi,
this is a test message
Output of MANTA_HOME/samples/src/sample/jms/queues/Talk/RunSample1.sh
> ./RunSample1.sh
*** MantaRay Layer
Name: m10.12.171.1226604 ***
MantaRay 2.0.1
initialization completed.
Start receiving
messages on queue "SampleQueue2".
Enter text to send
to queue "SampleQueue1".
Press Enter to send
each message.
Empty messages will
not be sent.
Typing 'exit' will
stop the program.
QUUERECEIVER>TOPICSENDER:
Hi, this is a test message
TOPICSENDER: Hi,
this is a test message
TOPICSENDER: Hi,
this is a test message
We have received the same message 3 times from the queue SampleQueue2.
This is because the MDB application in all the 3 instances of the
cluster is a subscriber to the topic and has received the message and
processed it. This results in 3 messages in the SampleQueue2.
This is an undesirable effect of deploying the application in a
cluster. In any enterprise scenario, this would be unacceptable.
In my next blog i will describe how mutually exclusive message
processing can be achieved. With few changes to the configuration and
MDB deployment descriptors and absoultely NO changes to the application
code, Generic JMS RA is capable of guranteeing that only one MDB
instance will process the message.
Known Issues and Limitations
All the MantaRay peers share the same configuration
file, so management of these peers may not be possible from the
MantaRay console because the RMI port will be the same. To use the
management console, please provide each instance (GlassFish instance)
with a separate MantaRay configuration file (mantaConfig JVM property
should point to the respective configuation file). Another drawback of
sharing the same configuration file is that the file based persistance
folder is also shared and this may not be desirable.
Resources
- Generic Resource Adapter for JMS community page -
download/documentation
- Generic Resource
Adapter for JMS user guide
- Project GlassFish
- MantaRay
- For additional
information on the 'RMPolicy' resource
adapter configuration attribute refer issue#5 in the Generic RA issue
tracker page. For documentation on usage of this attribute refer the
Generic Resource Adapter for JMS user guide.
