Binod's Blog

Community: Mobile & Embedded Archives

SailFin work and BTrace

Lately I have been a spending significant amount of time analyzing issues coming from sailfin performance team and system testing team. One of them was particularly tricky. Testing team was running a test 24x7 on a 10 instance sailfin cluster with CLB etc. There were 5 odd machines running many sipp clients pumping traffic to the sailfin. They started observing a memory leak and I went about debugging it.

jmap they produced (huge.. in GBs) showed the possibility of a leak of SipSession objects.

The problem with some memory leaks at times is the difficulty to confirm that the leak is in a particular data structure in a particular part of the code. Here also the case was the same.

So, I took the help of btrace.

I wrote a btrace script, without much difficulty. I ran the server under load for quite some time and then stopped the traffic so that SipSessions gets cleaned up. Then attached the btrace script to the running java process. I didnt have to compile the script just used the java file directly.

Here is the script, ConcurrentHashMap.java

package com.sun.btrace.binod;

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
import java.lang.reflect.Field;

@BTrace public class ConcurrentHashMapTrace {

  @OnMethod(
    clazz="java.util.concurrent.ConcurrentHashMap",
    method="put"
  ) 
  public static void onPut(java.util.concurrent.ConcurrentHashMap me, Object key, Object value) {
    //toString() of SipSessionDialogImpl contains the string SipSession.
    if (indexOf(str(value),"SipSession") > -1) {
        println(value);
        println(size(me));
    }
  }
}

Then I ran the command : btrace <pid> ConcurrentHashMapTrace.java. This attaches our btrace script to the JVM and waits.

And then sent just one INVITE message to the server. Btrace script printed the size of the concurrenthashmap that hold the sessions and it was more than I expected. That confirms the leak. I also used modified versions of the script to make sure that there is no other related object leak (eg: SipApplicationSessions).

After figuring out where the leak is, it was a matter of code inspection, further debugging and then checking in the fix..

Next, I am trying to see if System.gc is ever called during the test run so that we can try removing DisableExplicitGC flag from GC settings. Another btrace script....

SailFin at JavaOne

Tomorrow (Wednesday) morning at 9.30am, me and Kristoffer talk about SailFin architecture. The Venue is Hall E 134. This session gives the details about the project architecture, features and roadmap. Kristoffer and Ola have another session on Friday at 12.10 about End to End Communication Services with Java ME and Java EE. This is at Esplanade 303.

There is a hands on lab that demonstrates how to develop applications on SailFin. This is at 9.30am on Thursday at Hall E 130/131.

Then there is another hands on lab that shows performance analysis of SIP applications. This is at 1.00pm on Thursday, again at Hall E 130/131.

Many developers and architects from Sun and Ericsson are available in the SailFin booth in the Pavilion. Please do come and meet us!

FInally.... There is a Sun-Ericsson Application Competition that is launched this week. Please participate to win $5000 or a Sony Ericsson Phone.

Locating SIP servers using DNS in SailFin

As per RFC 3263, there are two new kind of records that are important for SIP. They are Naming Authority Pointer (NAPTR) Record and Service Record (SRV). In this example, NAPTR is used as a mechanism to specify the protocol (TCP or UDP) where as SRV records will be used locate the server and port from SailFin.

Example snippet from DNS zone file

_sip._udp.test.com. 3600 IN SRV 10 0 5090 sailfinserver.test.com.
_sip._tcp.test.com. 3600 IN SRV 10 0 5090 sailfinserver.test.com.
_sips._tcp.test.com. 3600 IN SRV 10 0 5061 sailfinserver.test.com.
_sips._udp.test.com. 3600 IN SRV 10 0 5061 sailfinserver.test.com.

;                  order pref flags service      regexp  replacement
test.com. IN NAPTR 0 0 "s" "SIPS+D2T" "" _sips._tcp.test.com.
test.com. IN NAPTR 1 0 "s" "SIP+D2T"  "" _sip._tcp.test.com.
test.com. IN NAPTR 2 0 "s" "SIP+D2U"  "" _sip._udp.test.com.

sailfinserver.test.com. IN A 192.168.1.100

The SRV records suggest that the port used by the server for TCP and UDB communication is 5090 and not the usual 5060 and the actual server is sailfinserver.test.com.

Finally we have an A record that resolves the sailfinserver.test.com to the computer's IP address.

SailFin uses dnsjava apis for name resolution. It allows the user to configure a JVM option called dns.server for specifying the name server. We could specify that as a jvm-option in domain.xml

If this is not specified, dnsjava also searches /etc/resolve.conf to find the name server. For experimenting that was better for me, as I always have a local name server running in the laptop. (For a moment forget Sip/SailFin, local name server (bind) performs much better for me than BSNL or opendns. That itself is a reason to run a local name server).

Here is my /etc/resolve.conf file content.

nameserver 127.0.0.1
nameserver 218.248.240.208
nameserver 4.2.2.1

Once these are configured, SailFin will be able to resolve the domain "test.com" and find correct SIP server, protocol, port etc. Here is the example of the INVITE message that I tried.

With these settings, SailFin will resolve "To" address to 192.168.1.100:5090 and will use TCP protocol.

Continue Reading...

This work is licensed under a Creative Commons License.