Skip to main content

TOTD #67: How to front-end a GlassFish Cluster with Apache + mod_jk on Mac OSX Leopard ?

Posted by arungupta on January 30, 2009 at 6:35 AM PST

GlassFish
provides support for High Availability by creating a cluster of server
instances and session state replication. This enhances the
scalability and availability of your application and is a critical
piece of decision making critieria when selecting an Application
Server.  href="http://developers.sun.com/appserver/reference/techart/glassfishcluster/">Clustering
in GlassFish Version 2 provides comprehensive introduction to
clustering, high availability and load balancing in GlassFish.



GlassFish provides out-of-the-box support for load-balancing href="http://docs.sun.com/app/docs/doc/819-3679/abdgs?a=view">HTTP(S),
JMS,
and RMI/IIOP
traffic and front-ended by Sun Java System Web Server, Apache Web
Server, and Microsoft IIS ( href="http://docs.sun.com/app/docs/doc/819-3679/gchvt?a=view">more
details here) using the href="https://glassfish.dev.java.net/javaee5/build/GlassFish_LB_Cluster.html">Load
Balancer plug-in. This plug-in however is not available for
Mac OS X and a popular technique used on that platform for front-ending
is to use Apache httpd
+ mod_jk.
This is exactly what this TOTD (Tip
Of style="font-weight: bold;">The style="font-weight: bold;">Day) is going to
describe.



This TOTD is going to explain how to front-end a 3-instance GlassFish
cluster with Apache httpd and mod_jk on Mac OS X.



This blog is using information from the following blogs:

  • href="http://blogs.sun.com/jluehe/entry/supporting_apache_loadbalancer_with_glassfish">How
    to Loadbalance GlassFish Cluster with Apache Loadbalancer
  • href="http://blogs.sun.com/jclingan/entry/glassfish_clustering_in_under_10">GlassFish
    Clustering in under 10 minutes
  • href="http://egopoly.com/2007/10/29/how-to-install-tomcat-mod_jk-on-mac-os-x-leopard/">How
    to install tomcat mod_jk on Mac OS X Leopard

And thanks to Vivek
and Shreedhar
for helping me understand the guts of GlassFish High Availability.



Without further ado, lets get started. The steps are slightly involving
so strap your seatbelts!

  1. First, lets create a 3-instance cluster following the
    screencast at href="http://blogs.sun.com/jclingan/entry/glassfish_clustering_in_under_10">GlassFish
    Clustering in under 10 minutes. Use the cluster name as
    "cluster1" and instance names as "instance1", "instance2", "instance3".
    The admin console will look like:



    href="http://blogs.sun.com/arungupta/resource/images/gfha-3instance-cluster.png"> style="border: 0px solid ; width: 600px; height: 275px;" alt=""
    src="http://blogs.sun.com/arungupta/resource/images/gfha-3instance-cluster.png">



    Deploy "clusterjsp" and make sure it works using port hopping as
    explained in the screencast. Click on each instance to identify their
    associated HTTP port.
  2. Define "jvmRoute" and "enableJK" properties on the newly
    created cluster as:


    style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
    cellpadding="2" cellspacing="2">
    ~/samples/v2/clustering/glassfish/bin
    >./asadmin
    create-jvm-options --target cluster1
    "-DjvmRoute=\${AJP_INSTANCE_NAME}"


    Command create-jvm-options executed successfully.

    ~/samples/v2/clustering/glassfish/bin > style="font-weight: bold;">./asadmin
    create-jvm-options --target cluster1
    "-Dcom.sun.enterprise.web.connector.enableJK=\${AJP_PORT}"

    Command create-jvm-options executed successfully.



    These properties are required to enable "stickiness" for "mod_jk". More
    details about how these properties are used internally is href="http://blogs.sun.com/jluehe/entry/supporting_apache_loadbalancer_with_glassfish">explained
    here.

  3. Configure the above system properties for each instance in
    the cluster as shown:


    style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
    cellpadding="2" cellspacing="2">
    ~/samples/v2/clustering/glassfish/bin
    >./asadmin
    create-system-properties --target instance1
    AJP_INSTANCE_NAME=instance1


    Command create-system-properties executed successfully.

    ~/samples/v2/clustering/glassfish/bin > style="font-weight: bold;">./asadmin
    create-system-properties --target instance1 AJP_PORT=9090

    Command create-system-properties executed successfully.

    ~/samples/v2/clustering/glassfish/bin > style="font-weight: bold;">./asadmin
    create-system-properties --target instance2 AJP_INSTANCE_NAME=instance2

    Command create-system-properties executed successfully.

    ~/samples/v2/clustering/glassfish/bin > style="font-weight: bold;">./asadmin
    create-system-properties --target instance2 AJP_PORT=9091

    Command create-system-properties executed successfully.

    ~/samples/v2/clustering/glassfish/bin > style="font-weight: bold;">./asadmin
    create-system-properties --target instance3 AJP_INSTANCE_NAME=instance3

    Command create-system-properties executed successfully.

    ~/samples/v2/clustering/glassfish/bin > style="font-weight: bold;">./asadmin
    create-system-properties --target instance3 AJP_PORT=9092

    Command create-system-properties executed successfully.



    Note the value of "AJP_PORT" property for each instance, this will be
    used for configuring "mod_jk" later. You may have to restart the
    cluster in order for these properties to be synchronized for each
    instance. This can be easily done using the admin console as explained
    in the screencast above.

  4. Install httpd: Mac OS X Leopard 10.5.6 comes bundled with
    Apache httpd
    2.2, so that's cool! Otherwise it can be downloaded from href="http://httpd.apache.org/download.cgi">httpd.apache.org.
    However in the pre-installed version there are some intricacies with
    directory
    names
    that are explained below.
  5. Lets install and configure "mod_jk" in "httpd".
    1. The mod_jk binaries for Mac OSX are supposedly available
      at  href="http://www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/macosx/">www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/macosx/.
      But installing the available module in httpd gives the following error:


      style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
      cellpadding="2" cellspacing="2">
      httpd: Syntax error on line 116 of
      /private/etc/apache2/httpd.conf: Cannot load
      /usr/libexec/apache2/mod_jk-1.2.25-httpd-2.2.4.so into server:
      dlopen(/usr/libexec/apache2/mod_jk-1.2.25-httpd-2.2.4.so, 10): no
      suitable image found.  Did
      find:\n\t/usr/libexec/apache2/mod_jk-1.2.25-httpd-2.2.4.so: mach-o, but
      wrong architecture
    2. The alternative is to build "mod_jk". Fortunately it
      turned out to be a straight forward process because of href="http://egopoly.com/2007/10/29/how-to-install-tomcat-mod_jk-on-mac-os-x-leopard/">this
      blog.
      1. Download href="http://tomcat.apache.org/download-connectors.cgi">latest
        Connectors source code ( href="http://mirrors.24-7-solutions.net/pub/apache/tomcat/tomcat-connectors/jk/source/jk-1.2.27/tomcat-connectors-1.2.27-src.tar.gz">version
        1.2.27). The file "BUILD.txt" (included in the source bundle)
        provide
        clear instructions to build and described below as well.
      2. Configure the build environment as shown:


        style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
        cellpadding="2" cellspacing="2">
        ~/workspaces/tomcat-connectors-1.2.27-src/native
        >./configure
        --with-apxs=/usr/sbin/apxs



        It shows the output as:


        style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
        cellpadding="2" cellspacing="2">
        .
        . .

        checking for target platform... unix

        no apache given

        no netscape given

        configure: creating ./config.status

        config.status: creating Makefile

        config.status: creating apache-1.3/Makefile

        config.status: creating apache-1.3/Makefile.apxs

        config.status: creating apache-2.0/Makefile

        config.status: creating apache-2.0/Makefile.apxs

        config.status: creating common/Makefile

        config.status: creating common/list.mk

        config.status: creating common/jk_types.h

        config.status: creating jni/Makefile

        config.status: creating common/portable.h

        config.status: executing depfiles commands
      3. Edit Edit "native/apache-2.0/Makefile.apxs" and add
        "-arch x86_64" as described href="http://egopoly.com/2007/10/29/how-to-install-tomcat-mod_jk-on-mac-os-x-leopard/">here.
        Please note that this string needs to be specified twice.
      4. Invoke "make" and "mod_jk.so" is generated in
        "native/apache-2.0" directory.
    3. Copy the generated "mod_jk.so" as:


      style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
      cellpadding="2" cellspacing="2">
      ~/workspaces/tomcat-connectors-1.2.27-src/native/apache-2.0
      >sudo cp mod_jk.so
      /usr/libexec/apache2/
    4. Load the "mod_jk" module in httpd by editing
      "/etc/apache2/httpd.conf". Please note another similar file exists in
      "/etc/httpd/httpd.conf" but ignore that one. Add the following as the
      last "LoadModule" line:


      style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
      cellpadding="2" cellspacing="2">
      LoadModule jk_module    
      libexec/apache2/mod_jk-1.2.25-httpd-2.2.4.so
    5. Configure "mod_jk" by adding the following lines
      immediately below the previously "LoadModule" line:


      style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
      cellpadding="2" cellspacing="2">
      JkWorkersFile
      /etc/apache2/worker.properties

      # Where to put jk logs

      JkLogFile /var/log/httpd/mod_jk.log

      # Set the jk log level [debug/error/info]

      JkLogLevel debug

      # Select the log format

      JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "

      # JkRequestLogFormat set the request format

      JkRequestLogFormat "%w %V %T"

      # Send all jsp requests to GlassFish

      JkMount /*.jsp loadbalancer



      The key lines in this configuration are the first and the last one.
      The first line informs "mod_jk" about the location of
      "worker.properties"
      file (explained later). The last line instructs to redirect only JSP
      requests. This will allow the static content such as images,
      text files, and media to be served from
      "httpd" itself.



      Also create the log directory specified in the configuration as:


      style="text-align: left; font-weight: bold; background-color: rgb(204, 204, 255); width: 100%;"
      cellpadding="2" cellspacing="2">
      sudo mkdir /var/log/httpd
    6. Create a new file "/etc/apache2/worker.properties".
      Again, this is not in "/etc/httpd" directory. Use the following
      contents:


      style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
      cellpadding="2" cellspacing="2">
      #
      Define 1 real worker using ajp13

      worker.list=loadbalancer

      # Set properties for instance1

      worker.instance1.type=ajp13

      worker.instance1.host=localhost

      worker.instance1.port=9090

      worker.instance1.lbfactor=50

      worker.instance1.cachesize=10

      worker.instance1.cache_timeout=600

      worker.instance1.socket_keepalive=1

      worker.instance1.socket_timeout=300

      # Set properties for instance2

      worker.instance2.type=ajp13

      worker.instance2.host=localhost

      worker.instance2.port=9091

      worker.instance2.lbfactor=50

      worker.instance2.cachesize=10

      worker.instance2.cache_timeout=600

      worker.instance2.socket_keepalive=1

      worker.instance2.socket_timeout=300

      # Set properties for instance3

      worker.instance3.type=ajp13

      worker.instance3.host=localhost

      worker.instance3.port=9092

      worker.instance3.lbfactor=50

      worker.instance3.cachesize=10

      worker.instance3.cache_timeout=600

      worker.instance3.socket_keepalive=1

      worker.instance3.socket_timeout=300



      worker.loadbalancer.type=lb

      worker.loadbalancer.balance_workers=instance1,instance2,instance3



      Read more about href="http://tomcat.apache.org/connectors-doc/reference/workers.html">worker.properties
      format. Essentially this file is used to load-balance a
      3-instance cluster and
      specify configuration values for each instance. Note that the value of
      "worker.instanceX.port" for instance X is exactly same as specified
      during instance configuration earlier.

  6. Copy "tomcat-ajp.jar" from the "lib" directory of the
    latest Tomcat
    5.5.x
    href="http://commons.apache.org/downloads/download_logging.cgi">commons-logging.jar
    ( href="http://www.gtlib.gatech.edu/pub/apache/commons/logging/binaries/commons-logging-1.1.1-bin.zip">version
    1.1.1), and  href="http://commons.apache.org/downloads/download_modeler.cgi">commons-modeler.jar
    ( href="http://apache.norhex.com/commons/modeler/binaries/commons-modeler-2.0.1.zip">version
    2.0.1) to GLASSFISH_HOME/lib. This is done as:


    style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
    cellpadding="2" cellspacing="2">
    ~/samples/v2/clustering/glassfish/lib
    >cp
    ~/tools/apache-tomcat-5.5.27/server/lib/tomcat-ajp.jar .


    ~/samples/v2/clustering/glassfish/lib > style="font-weight: bold;">cp
    ~/Downloads/commons-logging-1.1.1/commons-logging-1.1.1.jar .

    ~/samples/v2/clustering/glassfish/lib > style="font-weight: bold;">cp
    ~/tools/commons-modeler-2.0.1/commons-modeler-2.0.1.jar .



    You may have to restart cluster in order for these JARs to be loaded by
    each instance.

  7. An "httpd" instance is already running on port# 80 in my
    particular
    instance of Mac OS X. Instead of mangling with that, I decided to
    change the listening port for the new instance that will be spawn for
    out front-end. This can be easily done by editing
    "/etc/apache2/httpd.conf" and looking for lines similar to:


    style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
    cellpadding="2" cellspacing="2">
    #Listen 12.34.56.78:80

    Listen 80



    And change "Listen 80" to "Listen 81".

That completes the configuration, phew!



Lets start "httpd" as:


style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
cellpadding="2" cellspacing="2">
sudo httpd



The "httpd" logs are available in "/private/var/log/apache2". The
following message indicates a successful start of the web server:


style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
cellpadding="2" cellspacing="2">
. . .

[Thu
Jan 29 11:14:16 2009] [warn] Init: Session Cache is not configured
[hint: SSLSessionCache]

[Thu Jan 29 11:14:16 2009] [warn] No JkShmFile defined in httpd.conf.
Using default /usr/logs/jk-run

time-status

[Thu Jan 29 11:14:16 2009] [warn] No JkShmFile defined in httpd.conf.
Using default /usr/logs/jk-run

time-status

[Thu Jan 29 11:14:16 2009] [notice] Digest: generating secret for
digest authentication ...

[Thu Jan 29 11:14:16 2009] [notice] Digest: done

[Thu Jan 29 11:14:16 2009] [warn] pid file /private/var/run/httpd.pid
overwritten -- Unclean shutdow

n of previous Apache run?

[Thu Jan 29 11:14:16 2009] [notice] Apache/2.2.9 (Unix) mod_ssl/2.2.9
OpenSSL/0.9.7l DAV/2 mod_jk/1.

2.27 configured -- resuming normal operations



OK, now show time!



If everything has been configured properly as described above then
"http://localhost:81/clusterjsp/HaJsp.jsp" looks like:



alt=""
src="http://blogs.sun.com/arungupta/resource/images/gfha-hajsp-instance1.png">



Enter session attribute as "aaa" and value of the attribute as "111".
After you click on "ADD SESSION DATA" button, the updated page looks
like:



alt=""
src="http://blogs.sun.com/arungupta/resource/images/gfha-hajsp-instance1-session.png">



The highlighted part shows that the request is served from "instance1"
and recently added session data. Lets stop "instance1" and see if the
promise of high availability is fulfilled :)



Click on "Instances" tab, select "instance1" and click "Stop". The
admin console looks like:



href="http://blogs.sun.com/arungupta/resource/images/gfha-3instance-cluster-instance1-stopped.png"> style="border: 2px solid ; width: 600px; height: 266px;" alt=""
src="http://blogs.sun.com/arungupta/resource/images/gfha-3instance-cluster-instance1-stopped.png">



Notice "instance1" is shown as stopped. Clicking on "Reload Page" on
"http://localhost:81/clusterjsp/HaJsp.jsp"
shows:



alt=""
src="http://blogs.sun.com/arungupta/resource/images/gfha-hajsp-instance1-failover.png">





Aha!



Even though "instance1" is not runing the session data is still
available. And that is possible because of the seamless session
failover from primary ("instance1") to the replica partner
("instance2"). The highlighted part indicates that the request is now
indeed served by "instance2".



Here are some other useful links to consider:

  • High Availability Adminstration Guide ( href="http://docs.sun.com/app/docs/doc/819-3679/abdgs%3fa=view">online,
    pdf).
  • href="http://www.sun.com/offers/details/glassfish_HAref_config.html">GlassFish
    High Availability Reference Configurations.
  • href="http://www.sun.com/offers/details/glassfish_ha_virt_env.html">GlassFish
    High Availability Webinar.

Please leave suggestions on other TOTD (Tip Of The Day) that
you'd like to see.
A complete archive of all tips is available href="http://blogs.sun.com/arungupta/tags/totd">here.




Technorati: totd
glassfish
highavailability
clustering
loadbalancer
mod_jk
apache
httpd href="http://technorati.com/tag/mac">mac href="http://technorati.com/tag/leopard">leopard

Related Topics >>