Skip to main content

Distributed (JMS) messaging applications using GlassFish and MantaRay

Posted by rampsarathy on January 8, 2007 at 6:37 AM EST
Distributed (JMS) messaging applications using GlassFish and MantaRay

Distributed (JMS) messaging applications using GlassFish and MantaRay



   
Generic JMS RA (1.7) that is bundled with GlassFish V2 enables applications deployed on GlassFish to use almost any JMS 1.1 compliant message broker. This article talks about how applications (typically Message Driven Beans) deployed in a GlassFish  cluster can use MantaRay peer to peer architecture for  their JMS messaging needs using Generic resource adapter for JMS.
MantaRay is an open source enterprise-grade messaging middleware solution that is based on a peer-to-peer architecture. With its peer-to-peer topology, MantaRay eliminates the need for costly central messaging brokers.


picture1.png

                                          
                                                 Figure 1: GlassFish and MantaRay

The steps/configuration mentioned below are based on the following versions :

Project GlassFish V2 Build 30
https://glassfish.dev.java.net/downloads/v2-b30.html

MantaRay Java 2.0.1

Generic JMS RA 1.7
(Bundled with GlassFish, under GLASSFISH_HOME/lib/addons/resourceadapters/genericjmsra/genericra.rar) OR
https://genericjmsra.dev.java.net/files/documents/3308/43334/genericra.rar

Initial Setup

  • Refer to MantaRay documentation for installing MantaRay. Typically it involves extracting the zip file into a folder, which would be denoted  as MANTA_HOME in this document. 
    •           No changes are required to the configuration files for a basic setup.
             
           For the purpose of this document MantaRay  is installed under MANTA_HOME, GlassFish V2 is installed under GLASSFISH_HOME.   

      Install GlassFish application server and create a cluster, the domain that is used for this cluster creation should have been created using the cluster profile, the following steps briefly describe the tasks for 3 instance cluster creation.
             
            # NODE-AGENT CREATION
           
            GLASSFISH_HOME/bin/asadmin create-node-agent --user <adminUser>  --passwordfile <passwordFileLocation> --savemasterpassword=true agent1

            #START NODE-AGENT
            GLASSFISH_HOME/bin/asadmin start-node-agent --user  <adminUser> --passwordfile <passwordFileLocation> agent1
       
            # BEGIN CLUSTER CREATION
            GLASSFISH_HOME/bin/asadmin create-cluster --port <domainAdminPort> --user <adminUser> --passwordfile <passwordFileLocation> --autohadb=false                 cluster1

            # CREATE INSTANCE1

            GLASSFISH_HOME/bin/asadmin create-instance --port <domainAdminPort> --user <adminUser> --passwordfile <passwordFileLocation> --nodeagent ra-agent         --cluster cluster1 --systemproperties HTTP_LISTENER_PORT=1111:IIOP_LISTENER_PORT=33700 instance1

            # CREATE INSTANCE2
           GLASSFISH_HOME/bin/asadmin create-instance --user admin --port <domainAdminPort> --passwordfile <passwordFileLocation> --nodeagent agent1 --cluster         cluster1 --systemproperties HTTP_LISTENER_PORT=1112:IIOP_LISTENER_PORT=33701 instance2

            # CREATE INSTANCE3
            GLASSFISH_HOME/bin/asadmin create-instance --user admin --port <domainAdminPort> --passwordfile <passwordFileLocation> --nodeagent agent1 --cluster         cluster1 --systemproperties HTTP_LISTENER_PORT=1113:IIOP_LISTENER_PORT=33702 instance3


    • Modify the cluster1's classpath to add MantaRay  jars located in MANTA_HOME folder.  The asadmin GUI could be used to modify a domain's classpath. Open a browser and type the url of the application server admin GUI - https://hostname:adminport. Go to Configuration -> cluster1-config -> JVM Settings -> Path Settings . Add an entry for the jar files shown below in the classpath suffix. 
    MANTA_HOME/ext/antlr.jar   
    MANTA_HOME/manta.jar
    MANTA_HOME/ext/clink-1.5.0.jar
    MANTA_HOME/ext/commons-logging.jar
    MANTA_HOME/ext/log4j-1.2.8.jar

    • Modify the cluster1's JVM option to add MantaRay configuration file as a  JVM propert. The asadmin GUI could be used to do this. Open a browser and type the url of the application server admin GUI - https://hostname:adminport. Go to Configuration -> cluster1-config -> JVM Settings -> JVM options .
            -DmantaConfig=MANTA_HOME/config/default_config.xml      
    • Edit the server.policy file in the [GLASSFISH_HOME/domains/<domain>/config/] directory using your favourite text editor and add the following line to the default grant block.
                permission java.util.logging.LoggingPermission "control";
                permission java.util.PropertyPermission "*", "read,write";              

    Should you use an application client in your application, edit the application client's client.policy file in the [GLASSFISH_HOME/lib/appclient/client.policy] directory and add the following line to it.
                    permission javax.security.auth.PrivateCredentialPermission "javax.resource.spi.security.PasswordCredential * \"*\"","read";
    • Create a File system JNDI object store to bind ManataRay JMS administered objects.  The following code snippet creates a FS object store and binds the required MantaRay objects to the jndi tree.

    import javax.naming.*;
    import java.util.Properties;

    public class Main {
       
        public Main() {
        }
       
        public static void main(String[] args) {
        try {
                Properties props = new Properties();
                props.put("java.naming.factory.initial", "com.sun.jndi.fscontext.RefFSContextFactory");
                props.put("java.naming.provider.url", "file:/<objectStoreFolderName>");
                InitialContext jndiContext = new InitialContext(props);
                MantaQueue mq = new MantaQueue("SampleQueue1");
                MantaQueue outmq = new MantaQueue("SampleQueue2");          
                MantaTopic top = new MantaTopic("SampleTopic");
                MantaXAConnectionFactory confac = new MantaXAConnectionFactory();           
                jndiContext.rebind("SampleQueue1", mq);
                jndiContext.rebind("SampleQueue2", outmq);                       
                jndiContext.rebind("SampleTopic", top);           
                jndiContext.rebind("mantaxaconnectionfactory", confac);
               
            } catch (Exception e) {
                System.out.println("Could not create JNDI " + "context: " + e.toString());
                System.exit(1);
            }
                }
       
                }


    Configuring GlassFish cluster and  MantaRay


    picture.png


                      Figure 2 : Configuration

       
        The above configuration describes a simple MDB application that is configured to listen to a MantaRay topic SampleTopic, and when it receives a message from the topic  it sends the message to a queue SampleQueue2.
      MantaRay is a messaging middleware solution based on peer-peer architecture. MantaRay's support for JMS1.1 API has been exploited to achieve the above configuration. Applications deployed on GlassFish clusters (or instances) can use MantaRay's peer-peer messaging capabilities through standard JMS apis using the Generic JMS Resource Adapter. It supports both publish/subscribe and point-point messaging models. Please refer to MantaRay documentation for more details.

    JNDI object store : MantaRay administered objects are bound to a file system object store and generic JMS ra is configured to use this object store. Other JNDI stores (LDAP..) can also be used but their configuration is beyond the scope of this document. A simple standalone program (mentioned in the Initial Setup section) can be used to bind MantaRay administered objects to the JNDI tree.

    MantaRay configuration file : The default configuration file -MANTA_HOME/config/default_config.xml is used by all the MantaRay peers, since the default MantaRay configuration file is designed to be used by multiple peers there are no inherent port conflicts. The MantaRay peers would use ports in the range of 6600 to 6700 (default_config.xml). If you feel the need for configuring specific ports , then a separate configuration file has to be created for every peer. A typical use case for this would arise when you want to use the MantaRay management console, each peer would then require a unique RMI port.

    GlassFish Configuration :A GlassFish cluster is created with 3 (or N ) instances. The cluster configuration is modified to include MantaRay client jars in the classpath and also a JVM property is added for mantaConfig file. When an application (MDB) is deployed on the cluster it is deployed on all instances that belong to the cluster and also presents a homogeneous view to the user.  Every instance of the application in the cluster creates one MantaRay peer to consume or produce messages. The MDB uses the standard JMS 1.1 APIs to send/receive messages.  It also uses generic JMS resource adapter which provides connection pooling for the application.


    Note: Restart the cluster after completing all the changes.

    # Stopping the node agent stops all the instances in the cluster also.
    GLASSFISH_HOME/bin/asadmin stop-node-agent --user  <adminUser> --passwordfile <passwordFileLocation> agent1

    # Start the node agent with sync instances set to true, this would start the cluster with the updated configuration.
    GLASSFISH_HOME/bin/asadmin start-node-agent --user  <adminUser> --passwordfile <passwordFileLocation> --syncinstances=true agent1

    Configuring the Resource Adapter

    • Add ${appserver-install-dir}/bin to your PATH. The asadmin CLI command can be found at ${appserver-install-dir}/bin. In GlassFish a resource adapter configuration is used to specify the configuration of a resource adapter. Use the following command to create a resource adapter configuration for genericra, to configure it to work with MantaRay Java 2.0.1.
    asadmin create-resource-adapter-config --user <adminname> --password <admin password> --property SupportsXA=true:ProviderIntegrationMode=jndi:RMPolicy=OnePerPhysicalConnection:
    JndiProperties=java.naming.factory.initial\=com.sun.jndi.fscontext.RefFSContextFactory
    java.naming.provider.url\=file://space/mantarayobjects:LogLevel=FINEST genericra

    A brief description of the various properties used in the above command is explained below:
    SupportsXA
    Set the supports distributed transactions attribute to true. The level of transactional support the adapter provides -- none, local, or XA -- depends on the capabilities of the Enterprise Information System [EIS] being adapted. If an adapter supports XA transactions and this attribute is XA, the application can use distributed transactions to coordinate the EIS resource with JDBC and JMS resources.

    ProviderIntegrationMode
    Set the integration mode as JNDI. Two integration modes exist in the Generic Resource Adapter for JMS. The JNDI mode allows the resource adapter to use the administered objects published in the message provider's JNDI provider to integrate with the message provider.

    JndiProperties
    This property [comma-separated list of name-value pairs] specifies JNDI provider properties to be used for connecting to the JMS provider's JNDI. In our case, we set it to the JNDI configuration specified earlier.

    RMPolicy
    Some XAResource implementations such as IBM MQ Series, relies on a Resource Manager per Physical Connection and this causes issues when there is inbound and outbound communication to the same queue manager in a single transaction (For example, an MDB sends  a response to a destination). When RMPolicy is set to OnePerPhysicalConnection, the XAResource wrapper implementation's isSameRM in Generic JMS RA would check if both the XAResources use the same physical connection, before delegating to the wrapped objects. Ensure that this attribute is set to "OnePerPhysicalConnection" if the application uses XA.

    Deploying the Resource adapter

    • Download the Generic RA bits from the project site. With Glassfish V2, Generic RA is available out-of-the-box with the application server and you could choose to use the bundle resource adapter as well in the step below.
    • Deploy the resource adapter using the asadmin deploy command, as shown below. In the image above, see Generic JMS RA deployed in the application server.
                $ asadmin deploy --user admin --password adminadmin <location of the generic resource adapter rar file>
          

    Creating Connection Factories and Administered Objects in GlassFish.

    In order to configure a JMS Connection Factory, using the Generic Resource Adapter for JMS, a Connector connection pool and resource needs to be created in the application server, as shown below. In the image above, see inpool [pointing to Generic JMS RA and QCF] and jms/XAConFac [for inpool] created in the application server.

    Connector connection pool creation

    #Creates a Connection Pool called inpool and points to mantaxaconnectionfactory
    asadmin create-connector-connection-pool -- raname genericra connectiondefinition javax.jms.QueueConnectionFactory --transactionsupport  XATransaction --property ConnectionFactoryJndiName=mantaxaconnectionfactory inpool

    Connector resource creation

    #Creates a connector resource named jms/XAConFac and binds this resource to JNDI for applications to use.
    asadmin create-connector-resource --poolname inpool jms/XAConFac

    Admin Objects

    For JMS Destination Resources, an administered object needs to be created. In the image above, see jms/SampleQueue2 [pointing to Generic JMS RA and SampleQueue2] created in the application server.
    #Creates a javax.jms.Queue Administered Object and binds it to application server's JNDI tree at jms/SampleQueue2 and points to SampleQueue2
    asadmin create-admin-object --raname genericra --restype javax.jms.Queue --property DestinationJndiName=SampleQueue2 jms/SampleQueue2

    Component Deployment descriptors

    The deployment descriptors need to take into account the resource adapter and the connection resources that have been created. A sample sun-ejb-jar.xml for a Message Driven Bean that listens to a destination called SampleTopic in MantaRay, and publishes back reply messages to a destination resource named jms/SampleQueue2 is shown below.


    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE sun-ejb-jar PUBLIC "-//Sun Microsystems, Inc.//DTD Application Server 8.1 EJB 2.1//EN" "http://www.sun.com/software/appserver/dtds/sun-ejb-jar_2_1-1.dtd">
    <sun-ejb-jar>
      <enterprise-beans>
        <ejb>
          <ejb-name>MyBean</ejb-name>
          <jndi-name>sunMDB</jndi-name>
          <resource-ref>
            <res-ref-name>jms/MyQueueConnectionFactory</res-ref-name>
            <jndi-name>jms/XAConFac</jndi-name>
          </resource-ref>
          <resource-env-ref>
        <resource-env-ref-name>jms/OutQueue</resource-env-ref-name>
        <jndi-name>jms/SampleQueue2</jndi-name>
          </resource-env-ref>
          <bean-pool>
            <steady-pool-size>10</steady-pool-size>
            <resize-quantity>2</resize-quantity>
            <max-pool-size>30</max-pool-size>
            <pool-idle-timeout-in-seconds>60</pool-idle-timeout-in-seconds>
          </bean-pool>
          <mdb-resource-adapter>
            <resource-adapter-mid>genericra</resource-adapter-mid>
            <activation-config>
              <activation-config-property>
                <activation-config-property-name>DestinationType</activation-config-property-name>
                <activation-config-property-value>javax.jms.Topic</activation-config-property-value>
              </activation-config-property>
              <activation-config-property>
                <activation-config-property-name>MaxPoolSize</activation-config-property-name>
                <activation-config-property-value>30</activation-config-property-value>
              </activation-config-property>
              <activation-config-property>
                <activation-config-property-name>RedeliveryAttempts</activation-config-property-name>
                <activation-config-property-value>3</activation-config-property-value>
              </activation-config-property>
              <activation-config-property>
                <activation-config-property-name>RedeliveryInterval</activation-config-property-name>
                <activation-config-property-value>1</activation-config-property-value>
              </activation-config-property>
              <activation-config-property>
                <activation-config-property-name>ReconnectAttempts</activation-config-property-name>
                <activation-config-property-value>1000</activation-config-property-value>
              </activation-config-property>
              <activation-config-property>
                <activation-config-property-name>ReconnectInterval</activation-config-property-name>
                <activation-config-property-value>1</activation-config-property-value>
              </activation-config-property>
              <activation-config-property>
                <activation-config-property-name>DestinationJndiName</activation-config-property-name>
                <activation-config-property-value>SampleTopic</activation-config-property-value>           
              </activation-config-property>         
              <activation-config-property>
                <activation-config-property-name>ConnectionFactoryJndiName</activation-config-property-name>
                <activation-config-property-value>mantaxaconnectionfactory</activation-config-property-value>           
              </activation-config-property>    
            </activation-config>
          </mdb-resource-adapter>
        </ejb>
      </enterprise-beans>
    </sun-ejb-jar>

    The business logic encoded in Message Driven Bean could then lookup the configured QueueConnectionFactory/Destination resource
    to create a connection as shown below (in the onMessage(...) method).

    Context context = null;
    ConnectionFactory connectionFactory = null;
    logger.info("In PublisherBean.ejbCreate()");
    try {
        context = new InitialContext();
        queue = (javax.jms.Queue) context.lookup ("java:comp/env/jms/SampleQueue2");
        connectionFactory = (ConnectionFactory) context.lookup("java:comp/env/jms/MyQueueConnectionFactory");
        connection = connectionFactory.createConnection();
        QueueSession qss = connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
        QueueSender sender = qss.createSender(queue);
        TextMessage msg = qss.createTextMessage();
        msg.setText(txtmsg.getText());
        sender.send(msg);
        connection.close();

    } catch (Throwable t) {
        logger.severe("PublisherBean.ejbCreate:" + "Exception: " +
        t.toString());
    }

    Client

    The setup can be tested using a simple client sample that comes along with MantaRay


    We will use the MANTA_HOME/samples/src/sample/jms/topics/ReliableChat and MANTA_HOME/samples/src/sample/jms/queues/Talk as producer and consumer application respectively
    Complie the samples by executing the Compile.sh script that is provided under the above folders, this should generate the classes for these sample applications.

    To see it in action, modify the MANTA_HOME/samples/src/sample/jms/topics/ReliableChat/RunSample1.sh and ensure that we publish to the correct topic, "-t SampleTopic".
    Modify the MANTA_HOME/samples/src/sample/jms/queues/Talk/RunSample1.sh and ensure that we receive from the correct queue, "-qr SampleQueue2"

    Output of MANTA_HOME/samples/src/sample/jms/topics/ReliableChat/RunSample1.sh

    > ./RunSample1.sh
    *** MantaRay Layer Name: m10.12.171.1226603 ***
    MantaRay 2.0.1 initialization completed.

    Enter text messages to clients that subscribe to the SampleTopic topic.
    Press Enter to publish each message.
    Typing 'exit' will stop the program.

    Hi, this is a test message
    TOPICSENDER: Hi, this is a test message


    Output of MANTA_HOME/samples/src/sample/jms/queues/Talk/RunSample1.sh
     
    > ./RunSample1.sh
    *** MantaRay Layer Name: m10.12.171.1226604 ***
    MantaRay 2.0.1 initialization completed.

    Start receiving messages on queue "SampleQueue2".

    Enter text to send to queue "SampleQueue1".
    Press Enter to send each message.
    Empty messages will not be sent.
    Typing 'exit' will stop the program.
    QUUERECEIVER>TOPICSENDER: Hi, this is a test message
    TOPICSENDER: Hi, this is a test message
    TOPICSENDER: Hi, this is a test message


    We have received the same message 3 times from the queue SampleQueue2. This is because the MDB application in all the 3 instances of the cluster is a subscriber to the topic and has received the message and processed it. This results in 3 messages in the SampleQueue2.  This is an undesirable effect of deploying the application in a cluster. In any enterprise scenario, this would be unacceptable.

    In my next blog i will describe how mutually exclusive message processing can be achieved. With few changes to the configuration and MDB deployment descriptors and absoultely NO changes to the application code, Generic JMS RA is capable of guranteeing that only one MDB instance will process the message.

    Known Issues and Limitations


        All the MantaRay peers share the same configuration file, so management of these peers may not be possible from the MantaRay console because the RMI port will be the same. To use the management console, please provide each instance (GlassFish instance) with a separate MantaRay configuration file (mantaConfig JVM property should point to the respective configuation file). Another drawback of sharing the same configuration file is that the file based persistance folder is also shared and this may not be desirable.

    Resources

    Related Topics >>
    Blog Links >>