Jean-Francois Arcand's Blog

Preventing Rogue Applications to affect overall performance of Glassfish Prelude

An application server can get in a really bad shape when a rogue application/component gets deployed into it. How to prevent the situation using GlassFish Prelude? With the help of the bear, yes , you can minimize those rogues animals...

Just in time for the upcoming v3 Prelude release, Alexey and I have added a feature that add support for web applications isolation. You can isolate rogue components/applications by allocating a subset of the available threads or heap memory. I've already described the feature when Ajax based application are deployed in GlassFish, but this time you can apply the same technique for isolating applications from others' bad behavior. OK, but under which circumstances you want to do that? Well, there are several situations where you don't want your application to be affected by other deployed applications:

• Delayed response: when GlassFish is under load, you want to make sure your application will never get delayed by other applications who are doing expensive calculation
• All thread deadlocks: An application using JDBC might eventually eat a significant amount of GlassFish's WorkerThread because of the remote database. All Threads might ends up in a deadlock state where they are waiting for a response from the remote database. Worse, all your threads can lock, and there will be no available thread for servicing incoming requests
• And many other reasons...

Let's recap, from a previous blog, how it usually work in GlassFish when a request comes in. When requests comes in, the Grizzly HTTP module on which Prelude build on top of of, put them into a queue (see below)

When a WorkerThread becomes available, it get a request from the queue and execute it. When no thread are available, requests are waiting in the queue to be proceeded (in red below)

Now the normal behavior is to place the request at the end of the queue so every connection (or users) are equally/fairly serviced. Independently of how the request is executed, an application who needs to update its content real time (or very fast) might face a situation where the request is placed at the end of the queue, delaying the response from milliseconds to seconds. Hence, the usability of the application might significantly suffer if the server is getting under load and the queue is very large, or if a rogue/slow applications has already reserved the majority of the threads.

One solution isolate your application of the rogues application. How? By examining incoming requests and assign them to priority queues. Being able to prioritize requests might significantly improve the usability of an application and prevent rogue applications to affect its environment. Why? Because with resource isolation, you can make sure that specific requests will always be executed first and never placed into a queue, by either being placed at the head of the queue or by being executed by another queue:

As an example, request taking the form of /myApp/realTime might always gets executed before /rogueApp/

Want to try it? Then download GlassFish v3 Prelude and do the following (you can use the admin-gui if you don't want to edit the file manually):

% Install GlassFish
% Edit ${glassfish.home}/domains/domain1/config/domain.xml
% Under element http-listener, add the following line
  <property name="rcmSupport" value="true"/>
% Under jvm-options element, add:
  <jvm-options>-Dcom.sun.grizzly.rcm.policyMetric= 
  /yourApp/requestURI1|0.5,/yourApp/requestURI2|0.3</jvm-options>
% Start GlassFish

In the example above, Grizzly will reserve 50% of the thread to request taking the form of /yourApp/requestURI1, 30% for yourApp/requestURI2 and the remaining for all other incoming requests. Technically, it will means three queues will be created and Grizzly will dispatch the request to them based on the request URI

In conclusion, being able to isolate rogues applications/component on some policy rules (here request based) might significantly improve performance of your application. Have doubt? Just try it :-)

technorati: grizzly glassfish

Comments

• I would say it is fantastic :) Other good trick is to separate the server ports and then use the networks infrastructure to allocate more or less bandwith to specific ports (this is what I was planning to do.. including all sort of configuration mess).. Now I will include this Glassfish featuer as a new and much easier way to achieve the same results (I guess)...

  thanks a lot for this blog.. a new knowledge here .......

  Posted by: felipegaucho on October 03, 2008 at 01:36 AM

• Hi,
  have you any news about the JSR-284 (initially planned for jee6) which standardize a "Resource Consumption Management API"?
  a proposed final draft is present since mid-2007.
  do you plan to support it in glassfish v3?

  have you any news about advancement of this JSR?

  cheers,

  Charles GAY.

  Posted by: charles_gay on October 03, 2008 at 01:54 AM

• One wonders what ever happened to the Application Isolation API (JSR 121): http://jcp.org/en/jsr/detail?id=121

  Posted by: cowwoc on October 06, 2008 at 05:59 AM

• Felipe, Thanks!! Charles, I think someone from the GlasFish team is looking at that jsr. But on my side, I haven't yet add a chance to look at it. As for 121, I think that one is dead for awhile...A+ -- Jeanfrancois

  Posted by: jfarcand on October 06, 2008 at 07:58 AM

• Hi Jean-François,
  yes, opacity about JSR evolution is one of the main issues about JCP....
  but JSR-121 has got a final status, but not JSR-284( final draft since august 2007).
  it permits to everyone to follow one standard, and not reinvent the whell.
  it has been done by the jnode.org team (java os) for JSR-121.

  cheers,

  Charles.

  Posted by: charles_gay on October 06, 2008 at 01:00 PM

• Hey may be this is the wrong way to ask for yr help but its the only one i could think of. If u could just take a minute to look... We have a problem with glassfish/grezzly. It was some talk about it on SUN's forum, but it was NEVER answered. Plz contact me by email if possible. wind_57@mail.ru

  Posted by: wind_57 on October 08, 2008 at 12:59 AM

• Salut, please post your questions to users@grizzly.dev.java.net or users@glassfish.dev.java.net....I'm sure you will get a response really fast :-). A+

  Posted by: jfarcand on October 08, 2008 at 07:08 AM

This work is licensed under a Creative Commons License.