Skip to main content

TOTD #81: How to use nginx to load balance a cluster of GlassFish Gem ?

Posted by arungupta on April 30, 2009 at 5:47 AM PDT
cellspacing="2">
style="width: 121px; height: 32px;" alt=""
src="http://blogs.sun.com/arungupta/resource/ror/nginx.gif">
nginx
(pronounced as "engine-ex") is an open-source and high-performance HTTP
server. It provides the common features such as reverse proxying with
caching, load balancing, modular architecture using filters (gzipping,
chunked responses, etc), virtual servers, flexible configuration and
much more.



nginx is known for it's high performance and low resource consumption.
It's a fairly popular front-end HTTP server in the href="http://rubyonrails.org">Rails community
along with Apache, Lighttpd, and others. This TOTD ( style="font-weight: bold;">Tip style="font-weight: bold;">Of style="font-weight: bold;">The style="font-weight: bold;">Day) will show how
to install/configure nginx for load-balancing/front-ending a cluster of
Rails application running on href="http://rubyforge.org/projects/glassfishgem/">GlassFish
Gem.

  1. Download, build, and install nginx using the simple script ( href="http://snippets.dzone.com/posts/show/5668">borrowed
    from dzone):


    style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
    cellpadding="2" cellspacing="2">
    ~/tools
    > curl -L -O http://sysoev.ru/nginx/nginx-0.6.36.tar.gz

    ~/tools > tar -xzf nginx-0.6.36.tar.gz

    ~/tools > curl -L -O
    http://downloads.sourceforge.net/pcre/pcre-7.7.tar.gz

    ~/tools > tar -xzf pcre-7.7.tar.gz

    ~/tools/nginx-0.6.36 > ./configure
    --prefix=/usr/local/nginx --sbin-path=/usr/sbin --with-debug
    --with-http_ssl_module --with-pcre=../pcre-7.7

    ~/tools/nginx-0.6.36 > make

    ~/tools/nginx-0.6.36 > sudo make install

    ~/tools/nginx-0.6.36 > which nginx

    /usr/sbin/nginx



    OK, nginx is now roaring and can be verified by visiting
    "http://localhost" as shown below:



    alt=""
    src="http://blogs.sun.com/arungupta/resource/ror/nginx-welcome-page.png">

  2. Create a simple Rails scaffold as:


    style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
    cellpadding="2" cellspacing="2">
    ~/samples/jruby > style="font-weight: bold;">~/tools/jruby/bin/jruby -S rails
    runner

    ~/samples/jruby/runner >~/tools/jruby/bin/jruby
    script/generate scaffold runlog miles:float minutes:integer


    ~/samples/jruby/runner >sed
    s/'adapter: sqlite3'/'adapter: jdbcsqlite3'/
    <config/database.yml >config/database.yml.new


    ~/samples/jruby/runner >mv
    config/database.yml.new config/database.yml


    ~/samples/jruby/runner >~/tools/jruby/bin/jruby
    -S rake db:migrate
  3. Run this application using GlassFish Gem on 3 separate
    ports as:


    style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
    cellpadding="2" cellspacing="2">
    ~/samples/jruby/runner > style="font-weight: bold;">~/tools/jruby/bin/jruby -S
    glassfish

    Starting GlassFish server at: 192.168.1.145:3000 in development
    environment...

    Writing log messages to:
    /Users/arungupta/samples/jruby/runner/log/development.log.

    Press Ctrl+C to stop.



    The default port is 3000. Start the seond one by explicitly specifying
    the port using "-p" option ..


    style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
    cellpadding="2" cellspacing="2">
    ~/samples/jruby/runner > style="font-weight: bold;">~/tools/jruby/bin/jruby -S
    glassfish -p 3001

    Starting GlassFish server at: 192.168.1.145:3001 in development
    environment...

    Writing log messages to:
    /Users/arungupta/samples/jruby/runner/log/development.log.

    Press Ctrl+C to stop.



    and the last one on 3002 port ...


    style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
    cellpadding="2" cellspacing="2">
    ~/samples/jruby/runner > style="font-weight: bold;">~/tools/jruby/bin/jruby -S
    glassfish -p 3002

    Starting GlassFish server at: 192.168.1.145:3002 in development
    environment...

    Writing log messages to:
    /Users/arungupta/samples/jruby/runner/log/development.log.

    Press Ctrl+C to stop.



    On Solaris and Linux, you can run GlassFish as a daemon as well.

  4. Nginx currently uses a href="http://wiki.nginx.org/NginxFaq#What_algorithm_does_Nginx_use_to_load_balance.3F__Can_it_balance_based_on_connection_load.3F">simple
    round-robin algorithm. Other load balancers such as href="http://github.com/gnosek/nginx-upstream-fair/tree/master">nginx-upstream-fair
    (fair proxy) and href="http://github.com/ry/nginx-ey-balancer/tree/master">nginx-ey-balancer
    (maximum connections) are also available. The
    built-in algorithm will be used for this blog. Edit
    "/usr/local/nginx/conf/nginx.conf" to specify an href="http://wiki.nginx.org/NginxHttpUpstreamModule">upstream
    module which provides load balancing:
    1. Create a cluster
      definition by adding an upstream module ( href="http://wiki.nginx.org/NginxHttpUpstreamModule#upstream">configuration
      details) right before the "server" module:


      style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
      cellpadding="2" cellspacing="2">
      upstream
      glassfish {

              server
      127.0.0.1:3000;

              server
      127.0.0.1:3001;

              server
      127.0.0.1:3002;

          }



      The cluster specifies a bunch of GlassFish Gem instances running at the
      backend. Each server can be weighted differently as href="http://wiki.nginx.org/NginxHttpUpstreamModule#server">explained
      here. The port numbers must exactly match as those specified
      at the start up. The modified "nginx.conf" looks like:



      alt=""
      src="http://blogs.sun.com/arungupta/resource/ror/nginx-conf-upstream.png">



      The changes are highlighted on lines #35 through #39.

    2. Configure load balancing by specifying this cluster using
      "proxy_pass" directive as shown below:


      style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
      cellpadding="2" cellspacing="2">
      proxy_pass http://glassfish;



      in the "location" module. The updated "nginx.conf" looks like:



      alt=""
      src="http://blogs.sun.com/arungupta/resource/ror/nginx-conf-proxy-pass.png">



      The change is highlighted on line #52.

  5. Restart nginx by using the following commands:


    style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
    cellpadding="2" cellspacing="2">
    sudo
    kill -15 `cat /usr/local/nginx/logs/nginx.pid`

    sudo nginx

Now "http://localhost" shows the default Rails page as shown below:



alt=""
src="http://blogs.sun.com/arungupta/resource/ror/nginx-rails-welcome-page.png">



"http://localhost/runlogs"
now serves the page from the deployed Rails application.



Now
lets configure logging so that the upstream server IP address and port
are printed in the log files. In "nginx.conf", uncomment "log_format"
directive and add "$upstream_addr" variable as shown:


style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
cellpadding="2" cellspacing="2">
    log_format 
main  '$remote_addr - [$upstream_addr]
$remote_user [$time_local] $request '

                     
'"$status" $body_bytes_sent "$http_referer" '

                     
'"$http_user_agent" "$http_x_forwarded_for"';



    access_log 
logs/access.log  main;



Also
change the log format to "main" by uncommenting "access_log
logs/access.log main;" line as shown above (default format is
"combined"). Accessing "http://localhost/runlogs" shows the following
lines in "logs/access.log":


style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
cellpadding="2" cellspacing="2">
127.0.0.1 - [127.0.0.1:3000]
- [29/Apr/2009:15:27:57 -0700] GET
/runlogs/
HTTP/1.1 "200" 3689 "-" "Mozilla/5.0 (Macintosh;
U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like
Gecko) Version/3.2.1 Safari/525.27.1" "-"

127.0.0.1 - [127.0.0.1:3001]
- [29/Apr/2009:15:27:57 -0700] GET
/favicon.ico
HTTP/1.1 "200" 0 "http://localhost/runlogs/"
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us)
AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1"
"-"

127.0.0.1 - [127.0.0.1:3002]
- [29/Apr/2009:15:27:57 -0700] GET
/stylesheets/scaffold.css?1240977992
HTTP/1.1 "200" 889
"http://localhost/runlogs/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X
10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1
Safari/525.27.1" "-"



The
browser makes multiple requests (3 in this case) to load resources on a
page and they are nicely load-balanced on the cluster. If an instance
running on port 3002 is killed, then the access log show the entries
like:


style="text-align: left; background-color: rgb(204, 204, 255); width: 100%;"
cellpadding="2" cellspacing="2">
127.0.0.1 - [127.0.0.1:3000]
- [29/Apr/2009:15:28:53 -0700] GET /runlogs/ HTTP/1.1 "200" 3689 "-"
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us)
AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1"
"-"

127.0.0.1 - [127.0.0.1:3002,
127.0.0.1:3000
] - [29/Apr/2009:15:28:53 -0700] GET
/favicon.ico HTTP/1.1 "200" 0 "http://localhost/runlogs/" "Mozilla/5.0
(Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1
(KHTML, like Gecko) Version/3.2.1 Safari/525.27.1" "-"

127.0.0.1 - [127.0.0.1:3001]
- [29/Apr/2009:15:28:53 -0700] GET /stylesheets/scaffold.css?1240977992
HTTP/1.1 "200" 889 "http://localhost/runlogs/" "Mozilla/5.0 (Macintosh;
U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like
Gecko) Version/3.2.1 Safari/525.27.1" "-"



The second log line shows that server running on port 3002 did not
respond and so it automatically fall back to 3000, this is nice!



But this is
inefficient because a back-end trip is made even for serving a static
file ("/favicon.ico" and "/stylesheets/scaffold.css?1240977992"). This
can be easily solved by enabling Rails page caching as described href="http://brainspl.at/articles/2006/09/12/new-nginx-conf-with-rails-caching">here
and href="http://brainspl.at/articles/2007/01/03/new-nginx-conf-with-optimizations">here.



More options about logging are described in href="http://wiki.nginx.org/NginxHttpLogModule">NginxHttpLogModule
and upstream module variables are defined in href="http://wiki.nginx.org/NginxHttpUpstreamModule#Variables">NginxHttpUpstreamModule.



Here are some nginx resources:

Are you using nginx to front-end your GlassFish cluster ?



href="http://blog.headius.com/2009/04/apache-jruby-rails-glassfish-easy.html">Apache
+ JRuby + Rails + GlassFish = Easy Deployment! shows similar
steps if you want to front-end your Rails application running using
JRuby/GlassFish with Apache.



Hear all about it in href="http://blog.arungupta.me/2009/04/27/glassfish-and-netbeans-at-rails-conf-2009.aspx">Develop
with Pleasure, Deploy with Fun: GlassFish and NetBeans for a Better
Rails Experience session at href="http://en.oreilly.com/rails2009/">Rails Conf
next week.



Please leave suggestions on other TOTD ( style="font-weight: bold;">Tip style="font-weight: bold;">Of style="font-weight: bold;">The style="font-weight: bold;">Day) that
you'd like to see.
A complete archive of all tips is available href="http://blogs.sun.com/arungupta/tags/totd">here.




Technorati: href="http://technorati.com/tags/rubyonrails">rubyonrails
glassfish
v3 href="http://technorati.com/tags/gem">gem href="http://technorati.com/tags/jruby">jruby href="http://technorati.com/tags/nginx">nginx href="http://technorati.com/tags/loadbalancing">loadbalancing
clustering

Related Topics >>