Binod's Blog: Open Source Archives

This time, my work had about 800 lines changes/new lines spanning 16 files and was implementing two RFCs. All that was plain simple java code and not very complicated, but still the worry I always had about presenting the changes to the reviewers (in different geos) came up on my head. It is still my code and not their code. It should be as easy as possible for them to understand the changes and review them.

Fortunately atlassian folks helped me here. They setup Crucible 1.6 beta for SailFin and gave us a page for signing up. It solved most of my worries on sending these changes for review.

Crucible gives a web page where you can upload your patches. (cvs -upN diff).
Crucible converts your patch to distinct files, color coding for the diffs etc
Comments can be added right into the code itself

Here is a copy of my changeset in crucible demo project, currently under review.

Crucible has been around for a while... And if you havent yet checked out, please do.

SailFin work and BTrace

Posted by binod on June 12, 2008 at 05:37 AM | Permalink | Comments (7)

Lately I have been a spending significant amount of time analyzing issues coming from sailfin performance team and system testing team. One of them was particularly tricky. Testing team was running a test 24x7 on a 10 instance sailfin cluster with CLB etc. There were 5 odd machines running many sipp clients pumping traffic to the sailfin. They started observing a memory leak and I went about debugging it.

jmap they produced (huge.. in GBs) showed the possibility of a leak of SipSession objects.

The problem with some memory leaks at times is the difficulty to confirm that the leak is in a particular data structure in a particular part of the code. Here also the case was the same.

So, I took the help of btrace.

I wrote a btrace script, without much difficulty. I ran the server under load for quite some time and then stopped the traffic so that SipSessions gets cleaned up. Then attached the btrace script to the running java process. I didnt have to compile the script just used the java file directly.

Here is the script, ConcurrentHashMap.java

package com.sun.btrace.binod;

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
import java.lang.reflect.Field;

@BTrace public class ConcurrentHashMapTrace {

  @OnMethod(
    clazz="java.util.concurrent.ConcurrentHashMap",
    method="put"
  ) 
  public static void onPut(java.util.concurrent.ConcurrentHashMap me, Object key, Object value) {
    //toString() of SipSessionDialogImpl contains the string SipSession.
    if (indexOf(str(value),"SipSession") > -1) {
        println(value);
        println(size(me));
    }
  }
}

Then I ran the command : btrace <pid> ConcurrentHashMapTrace.java. This attaches our btrace script to the JVM and waits.

And then sent just one INVITE message to the server. Btrace script printed the size of the concurrenthashmap that hold the sessions and it was more than I expected. That confirms the leak. I also used modified versions of the script to make sure that there is no other related object leak (eg: SipApplicationSessions).

After figuring out where the leak is, it was a matter of code inspection, further debugging and then checking in the fix..

Next, I am trying to see if System.gc is ever called during the test run so that we can try removing DisableExplicitGC flag from GC settings. Another btrace script....

Powered by
Movable Type 3.01D

java.net RSS Feeds