This blog is a follow up to the excellent blog Sreeram did on Resource Consumption Management (RCM) in GlassFish. I strongly recommend you read it before this one.

The Grizzly's Application Resources Allocation (ARA) extension is an implementation of a Resource Consumption Management (RCM) system. RCM system are allowing you to enable virtualization of system resources per web application, similar to Solaris 10 zone or the outcome of the upcoming JSR 284.

ARA has been implemented on top of Grizzly's Pipeline and Algorithm interface and a new basic rule-based filtering mechanism to manipulate tcp request URLs on the fly (browse the code here). For example, you can easily add a rule to rewrite a tcp request, like mod_rewrite does in Apache.

Grizzly's ARA currently supports two rule that can be applied to a tcp request:

• Reserve a specific percentage of the available heap memory.
• Reserve a specific percentage of the available threads.

You can configure the rule semantic using two kind of allocation policy: reservation or ceiling.

Reservation
Lets say there are 100 threads in the Grizzly request processing thread pool(Pipeline). For example, we reserve 30% of the Pipeline's threads to App A and 65% to App B. Now that leaves only 5% to other applications. So even if the other 95% threads are idle, we delay requests until the 5 unreserved threads can service requests. If the reservations are speaking for all available resources, then we may reject the request.

Ceiling
Ceiling has a slightly different behavior. If there are more than 30 requests to A, even if the threads set aside for B and others are idle, we delay processing the requests to A, until A's utilized thread percentage falls below 30%. Here we may delay requests until resources become available.

Like many Grizzly and WebContainer extension, the ARA mechanism is disabled by default, and will be considered to become an official feature when the GlassFish.next discussion starts. Since this is an experimental feature right now, we also want to get community feedback and contribution before rolling it in as a supported feature. Thus, for now, you gonna need to edit directly GlassFish's domain.xml file.

To enable ARA, you first need to tell Grizzly to use the ARA specialized Pipeline:

<jvm-options>-Dcom.sun.enterprise.web.connector.grizzly.pipelineClass=
           com.sun.enterprise.web.ara.IsolationPipeline</jvm-options>

Then, you need to define the rule(s) you want to be applied to the tcp request:

<jvm-options>-Dcom.sun.enterprise.web.ara.rules=
      com.sun.enterprise.web.ara.rules.ThreadRatioRule,
      com.sun.enterprise.web.ara.rules.HeapRatioRule</jvm-options>

I usually apply only one rule at the time. Note that it is fairly easy to add a new Rule to the rule-based engine. All you need to do is to implement the Rule interface, and tell the rule-based engine to load it at startup using the above property. Your rule.callable() implementation will have to implement the following contract:

• Rule_OK: the rule has been successfully applied and the request can be executed. The rule result will be cached so the next request will be serviced directly, without paying the price of the rule based engine execution.
• Rule_DELAY: the rule has been successfully applied but the request execution must be delayed.
• Rule_OK_NOCACHE: the rule has been successfully applied and the request can be executed, but the rule must not be cached so next time the request is coming, the rule will be executed.

The ARA implementation will invoke rule.attach() with an NIO ByteBuffer which position and limit will delimitate the HTTP request context root, which will be used to execute the rule. Take a look here for an example

Then you need to configure the rule semantic allocation policy:

<jvm-options>-Dcom.sun.enterprise.web.ara.policyMethod=reserve</jvm-options>

or

<jvm-options>-Dcom.sun.enterprise.web.ara.policyMethod=ceiling</jvm-options>

For the ceiling allocation policy, you can define the time the request will be delayed when no resource are available by adding (the value in milliseconds):

<jvm-options>-Dcom.sun.enterprise.web.ara.delay=30000</jvm-options>

Then finaly, define the application set on which rule(s) will be applied:

<jvm-options>-Dcom.sun.enterprise.web.ara.policyMetric=appA|0.5,appB|0.4</jvm-options>

The allowed values are application name, the token '|', then the allowed ratio (between 0 and 1) e.g. ([yourApp|ratio] separated by a ',')

Ouf...can't wait for the admin team to implement something easier to configure :-). Let me know if the steps are unclear (which I suspect they are). Once all the jvm-options has been entered, launch GlassFish. Grizzly will replace its default Pipeline with the above defined ARA, and starts the rule-base engine.

Now why not implementing the same behavior using a Servlet's Filter or a Catalina's Valve? Because performance wise, you don't want to parse the request and then do RCM. Doing such filtering at the bytes level, although it complexity the rule implementation, reduce the performance degradation. You certainly don't want you server to significantly slow down when RCM is enabled.

Interested by RCM and ARA? Have some rules in mind or rule implementation ready? Or rules recommendations? Feel free to post them here!
technorati: grizzly resource consumption management glassfish