Binod's Blog

Binod P.G is a senior staff engineer in the Java Web Services division at Sun Microsystems. He is an architect in the Application Server development team and is working on Project SailFin. He is also a co-specification lead of the Java EE Connector Architecture 1.6 Expert Group and a member of JDBC 4.0 expert group. In the past, he was involved in the development of many areas of the glassfish applicaton server, including Java EE Service Engine, Server Startup, Connector 1.5, JDBC, Connection Pool and JMS provider integration. He is also one of the owners of Generic Resource Adapter for JMS project. Prior to joining Sun in 2000, he has worked on a number of server side software technologies including IMS PL/1 programs in IBM Mainframes and internet projects in Microsoft IIS.

SailFin work and BTrace

Lately I have been a spending significant amount of time analyzing issues coming from sailfin performance team and system testing team. One of them was particularly tricky. Testing team was running a test 24x7 on a 10 instance sailfin cluster with CLB etc. There were 5 odd machines running many sipp clients pumping traffic to the sailfin. They started observing a memory leak and I went about debugging it.

jmap they produced (huge.. in GBs) showed the possibility of a leak of SipSession objects.

The problem with some memory leaks at times is the difficulty to confirm that the leak is in a particular data structure in a particular part of the code. Here also the case was the same.

So, I took the help of btrace.

I wrote a btrace script, without much difficulty. I ran the server under load for quite some time and then stopped the traffic so that SipSessions gets cleaned up. Then attached the btrace script to the running java process. I didnt have to compile the script just used the java file directly.

Here is the script, ConcurrentHashMap.java

package com.sun.btrace.binod;

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
import java.lang.reflect.Field;

@BTrace public class ConcurrentHashMapTrace {

  @OnMethod(
    clazz="java.util.concurrent.ConcurrentHashMap",
    method="put"
  ) 
  public static void onPut(java.util.concurrent.ConcurrentHashMap me, Object key, Object value) {
    //toString() of SipSessionDialogImpl contains the string SipSession.
    if (indexOf(str(value),"SipSession") > -1) {
        println(value);
        println(size(me));
    }
  }
}

Then I ran the command : btrace <pid> ConcurrentHashMapTrace.java. This attaches our btrace script to the JVM and waits.

And then sent just one INVITE message to the server. Btrace script printed the size of the concurrenthashmap that hold the sessions and it was more than I expected. That confirms the leak. I also used modified versions of the script to make sure that there is no other related object leak (eg: SipApplicationSessions).

After figuring out where the leak is, it was a matter of code inspection, further debugging and then checking in the fix..

Next, I am trying to see if System.gc is ever called during the test run so that we can try removing DisableExplicitGC flag from GC settings. Another btrace script....

SailFin at JavaOne

Tomorrow (Wednesday) morning at 9.30am, me and Kristoffer talk about SailFin architecture. The Venue is Hall E 134. This session gives the details about the project architecture, features and roadmap. Kristoffer and Ola have another session on Friday at 12.10 about End to End Communication Services with Java ME and Java EE. This is at Esplanade 303.

There is a hands on lab that demonstrates how to develop applications on SailFin. This is at 9.30am on Thursday at Hall E 130/131.

Then there is another hands on lab that shows performance analysis of SIP applications. This is at 1.00pm on Thursday, again at Hall E 130/131.

Many developers and architects from Sun and Ericsson are available in the SailFin booth in the Pavilion. Please do come and meet us!

FInally.... There is a Sun-Ericsson Application Competition that is launched this week. Please participate to win $5000 or a Sony Ericsson Phone.

June 2008

Search this blog:

Categories

Archives

June 2008
May 2008
February 2008
December 2006
November 2006
July 2006
May 2006
November 2005
September 2005

Recent Entries

SailFin work and BTrace

SailFin at JavaOne

Locating SIP servers using DNS in SailFin

This work is licensed under a Creative Commons License.