Skip to main content

SailFin work and BTrace

Posted by binod on June 12, 2008 at 5:37 AM PDT

Lately I have been a spending significant amount of time analyzing issues coming from sailfin performance team and system testing team. One of them was particularly tricky. Testing team was running a test 24x7 on a 10 instance sailfin cluster with CLB etc. There were 5 odd machines running many sipp clients pumping traffic to the sailfin. They started observing a memory leak and I went about debugging it.

jmap they produced (huge.. in GBs) showed the possibility of a leak of SipSession objects.

The problem with some memory leaks at times is the difficulty to confirm that the leak is in a particular data structure in a particular part of the code. Here also the case was the same.

So, I took the help of btrace.

I wrote a btrace script, without much difficulty. I ran the server under load for quite some time and then stopped the traffic so that SipSessions gets cleaned up. Then attached the btrace script to the running java process. I didnt have to compile the script just used the java file directly.

Here is the script, ConcurrentHashMap.java

package com.sun.btrace.binod;

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
import java.lang.reflect.Field;

@BTrace public class ConcurrentHashMapTrace {

  @OnMethod(
    clazz="java.util.concurrent.ConcurrentHashMap",
    method="put"
  )
  public static void onPut(java.util.concurrent.ConcurrentHashMap me, Object key, Object value) {
    //toString() of SipSessionDialogImpl contains the string SipSession.
    if (indexOf(str(value),"SipSession") > -1) {
        println(value);
        println(size(me));
    }
  }
}

Then I ran the command : btrace ConcurrentHashMapTrace.java.
This attaches our btrace script to the JVM and waits.

And then sent just one INVITE message to the server. Btrace script printed the size of the concurrenthashmap that hold the sessions and it was more than I expected. That confirms the leak. I also used modified versions of the script to make sure that there is no other related object leak (eg: SipApplicationSessions).

After figuring out where the leak is, it was a matter of code inspection, further debugging and then checking in the fix..

Next, I am trying to see if System.gc is ever called during the test run so that we can try removing DisableExplicitGC flag from GC settings. Another btrace script....

Related Topics >>

Comments

Tom, We can provide ready-made btrace script files that probe specific areas of the server. Users can use them as a starting point to produce information that will help us understand the problems more clearly. The output is somewhat a mathematical evidence of the state of the server. I think Mahesh and others in GF team is already working on something similar.

Markus, Thanks for the comment. Many tools like NB profiler (http://profiler.netbeans.org/) do support heap dump analysis. I have used many of these in the past. The thing I liked about Btrace is the ability to probe specific areas of code by just attaching to the running sailfin server.

Great post Binod. I don't know much about Btrace. Are there things we can put in GF and SailFin to make them BTrace and easier to debug and hence more serviceable.

Hi, Nice post ! I always wanted to give btrace a try, and I even had downloaded it already. I guess I have to take a look at it soon. Regarding the analysis of heap dumps I can highly recommend you the Eclipse Memory Analyzer (http://www.eclipse.org/mat). With the Memory Analyzer you often can find the cause for a memory leak without the need for a stack trace. Regards, Markus ( Java Performance Blog)

One thing painfully obvious about this approach is that one clearly needs to understand the actual implementation and the execution flow. Not very practical and maintainable.

Would it not have been much easier to just ** observe ** the SipSessions created and then to watch their life cycle from within an application management console or memory profiler. If you are going to do it the way described above you would be much better off just firing up a debugger but then that would not be fun and leisurely.


JXInsight 5.5.EA.9 - Object Spotting
http://blog.jinspired.com/?p=148

William

yardus, I will contact you offline.

Yes, BTrace is a marvelous tool. Big kudos to Sundar. I added an experimental support to NetBeans so you can create and run BTrace scripts from within NB with all the conveniences it provides. Also, there is a module for VisualVM allowing you to run a pre-packaged BTrace script against a particular process. They are both available from BTrace CVS. binod, could you introduce me to guys working on BTrace scripts for GF/SailFin?