jmap they produced (huge.. in GBs) showed the possibility of a leak of SipSession objects.

The problem with some memory leaks at times is the difficulty to confirm that the leak is in a particular data structure in a particular part of the code. Here also the case was the same.

So, I took the help of btrace.

I wrote a btrace script, without much difficulty. I ran the server under load for quite some time and then stopped the traffic so that SipSessions gets cleaned up. Then attached the btrace script to the running java process. I didnt have to compile the script just used the java file directly.

Here is the script, ConcurrentHashMap.java

package com.sun.btrace.binod;

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;
import java.lang.reflect.Field;

@BTrace public class ConcurrentHashMapTrace {

  @OnMethod(
    clazz="java.util.concurrent.ConcurrentHashMap",
    method="put"
  ) 
  public static void onPut(java.util.concurrent.ConcurrentHashMap me, Object key, Object value) {
    //toString() of SipSessionDialogImpl contains the string SipSession.
    if (indexOf(str(value),"SipSession") > -1) {
        println(value);
        println(size(me));
    }
  }
}

Then I ran the command : btrace <pid> ConcurrentHashMapTrace.java. This attaches our btrace script to the JVM and waits.

And then sent just one INVITE message to the server. Btrace script printed the size of the concurrenthashmap that hold the sessions and it was more than I expected. That confirms the leak. I also used modified versions of the script to make sure that there is no other related object leak (eg: SipApplicationSessions).

After figuring out where the leak is, it was a matter of code inspection, further debugging and then checking in the fix..

Next, I am trying to see if System.gc is ever called during the test run so that we can try removing DisableExplicitGC flag from GC settings. Another btrace script....

There is a hands on lab that demonstrates how to develop applications on SailFin. This is at 9.30am on Thursday at Hall E 130/131.

Then there is another hands on lab that shows performance analysis of SIP applications. This is at 1.00pm on Thursday, again at Hall E 130/131.

Many developers and architects from Sun and Ericsson are available in the SailFin booth in the Pavilion. Please do come and meet us!

FInally.... There is a Sun-Ericsson Application Competition that is launched this week. Please participate to win $5000 or a Sony Ericsson Phone.

Example snippet from DNS zone file

_sip._udp.test.com. 3600 IN SRV 10 0 5090 sailfinserver.test.com.
_sip._tcp.test.com. 3600 IN SRV 10 0 5090 sailfinserver.test.com.
_sips._tcp.test.com. 3600 IN SRV 10 0 5061 sailfinserver.test.com.
_sips._udp.test.com. 3600 IN SRV 10 0 5061 sailfinserver.test.com.

;                  order pref flags service      regexp  replacement
test.com. IN NAPTR 0 0 "s" "SIPS+D2T" "" _sips._tcp.test.com.
test.com. IN NAPTR 1 0 "s" "SIP+D2T"  "" _sip._tcp.test.com.
test.com. IN NAPTR 2 0 "s" "SIP+D2U"  "" _sip._udp.test.com.

sailfinserver.test.com. IN A 192.168.1.100

The SRV records suggest that the port used by the server for TCP and UDB communication is 5090 and not the usual 5060 and the actual server is sailfinserver.test.com.

Finally we have an A record that resolves the sailfinserver.test.com to the computer's IP address.

SailFin uses dnsjava apis for name resolution. It allows the user to configure a JVM option called dns.server for specifying the name server. We could specify that as a jvm-option in domain.xml

If this is not specified, dnsjava also searches /etc/resolve.conf to find the name server. For experimenting that was better for me, as I always have a local name server running in the laptop. (For a moment forget Sip/SailFin, local name server (bind) performs much better for me than BSNL or opendns. That itself is a reason to run a local name server).

Here is my /etc/resolve.conf file content.

nameserver 127.0.0.1
nameserver 218.248.240.208
nameserver 4.2.2.1

Once these are configured, SailFin will be able to resolve the domain "test.com" and find correct SIP server, protocol, port etc. Here is the example of the INVITE message that I tried.

With these settings, SailFin will resolve "To" address to 192.168.1.100:5090 and will use TCP protocol.

1. sip-proxy.war : A sip application as explained in Sip Proxy example of sailfin.
2. xyz.zone which contains all the DNS records explained above.
3. named.conf of my bind DNS.
4. uac.xml and uas.xml files for running the sipp tests (I have hard coded the test.com in the files)

Steps to try DNS in sailfin.

1. Setup and start the nameserver (bind). The example zip file contains named.conf and the domain zone file.
2. After starting the DNS, dig it and see, if it is working (eg: dig @127.0.0.1 NAPTR test.com). Make sure that the ANSWER from DNS contains the SRV records.
3. Configure either the dns.server jvm-option in sailfin or /etc/resolve.conf
4. Start a SailFin server and deploy the sip proxy servlet.
5. Use Sipp to execute a proxy scenario using the uac.xml and uas.xml. (The only change from the proxy example will be to use the sip proxy files.

A lot of work happened in GlassFish V2 for integrating JBI. Mainly these are in the areas of administration, clustering and runtime. Two JBI components (Java EE Service Engine and HTTP/SOAP Binding component) also will be available in GlassFish as system components.

Administration Experience

Apart from all the tools that was available in earlier JBI runtime, now it can be administered using Glassfish's own asadmin command line interface. See below Jane's CLI command finder suggesting a number JBI commands.

A complete list of commands and more details are available in JBI wiki page. GlassFish GUI has the capability to manage the JBI artifacts also. The picture shows a snapshot I took some time back.

Apart from these, composite applications (service assemblies) can be dropped in to autodeploy folder of the GlassFish application server and they will be deployed into the JBI runtime in appserver.

Clustering

JBI runtime has been enhanced to adhere to the appserver clustering architecture. Each instance in the appserver cluster will also have a JBI runtime in it. That means a JBI component or service assembly can be deployed to the GlassFish appserver cluster. It will be activated dynamically in each server instance. The components (similar to java ee containers) should implement their own load balancing and high availability mechanism. There are a number of components that are being developed as part of this open source project.

Runtime

JBI is integrated with appserver as a pre-configured lifecycle module. That means whenever an appserver instance starts up, the JBI runtime will be available. There is no user interaction required to configure or start the JBI runtime. It is just like any other service of appserver.

Java EE Service Engine act as the bridge between Java EE applications and JBI. A number of enhancements and performance improvements has been implemented in the Java EE Service Engine.

A Java EE application archive (ear/war/jar) can be packaged in a JBI composite application and deployed as one single entity.

Service units belonging to other JBI components and Java EE applications can share the same transaction and security contexts.

Summary

From user's point of view, glassfish now supports JBI. It means that glassfish now supports composite applications (Service assemblies) and a Java EE application can be part of one such composite application. A host of netbeans based tools and other JBI components will be available. So, application integration is made simpler (I know, still it is a big challenge).

For more information take a look at user-experience presentation slides and one pager. Also JBIWiki has a host of JBI related information.

Enjoy GlassFish!

Bobby is located in Santa Clara and I am located in Bangalore office of Sun Microsystems. Bobby setup a VNC Server on the laptop during his night so that I can look at whats happening during my day. Global engineering certainly certainly has advantages :-)

Debugging spell was quite longer than usual. Finally, we found that the issue was because of a firewall setting in the norton windows firewall.

As explained here symantec firewall has a setting called "Default Inbound loopback" which is configured to allow only 127.0.0.1. When this configuration is changed to allow any IP address (well bobby was using DHCP) the server started working.

This is not the first firewall related issue glassfish team has faced. Another one is with the Windows java.nio Selector implementation. Windows selector implementation in JDK use a loopback socket to handle the wakeup mechanism. But some firewalls are particularly strict against loopback sockets. Typically the issue will surface with the following stacktrace

java.io.IOException: Unable to establish loopback connection
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:106)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.PipeImpl.(PipeImpl.java:122)
at sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:27)

There are so many reports about this issue. Here are some links where people faced it outside glassfish [apache, limeware]

A neat explanation about this issue can be found in this sun forum link on socket programming.

Thats all for today. Now that I have started working from home for 2 days a week, hopefully I will get time to write next blog soon.

Thanks to Bobby for his excellent support on debugging some of these issues.

The Sun booth will demonstrate a Java EE 5 application getting developed on netbeans and deployed on GlassFish. We had a dry run of the app (Sahoo wrote it) at Sun campus this week and it is indeed a nice and simple explanation of what netbeans and glassfish is capable of.

Come, interact with the GlassFish engineers in the booth and pick up a glassfish CD.

For those who are interested in what foss.in is all about, see the event history.

Updates:

Sahoo blogged about the demo. Take a look.

Siva and Prasad has blogged [1, 2] about the event also.

Ok, enough of theory, lets get into some action. Below is the architecture diagram of an example application.

As explained in the diagram, the client application sends a message to the Queue which is picked up by the MDB. Upon receiving that message, by using the Java EE Service Engine, the MDB contacts the JBI NMR for the service endpoint exposed by the BPEL process. When MDB executes the BPEL application (hosted by BPEL service engine) BPEL app will contact JBI NMR for finding its partner services. In our BPEL example, there is only one partner service and it is a stateless session bean (JSR 109 webservice) hosted in the same appserver JVM. The JSR 109 webservice's endpoint is enabled in JBI NMR by Java EE Service Engine when the stateless session bean gets deployed. This SLSB uses new Java EE persistence apis to access the database.

In fact, Java EE SDK and Netbeans Enterprise Pack hides many complexities from the developer. Thus a developer's view of the composite application is much simple and as given below. I just hid engines and NMR and made a simpler diagram.

As explained in the diagram above, the MDB just uses JAXWS client apis to access the BPEL process. The BPEL process then invokes the Java EE webservice. The SLSB(Java EE webservice) uses persistence API to access the database. So, a developer just need to code the MDB, the stateless session bean and persistence code and design a BPEL process using these in the BPEL designer.

Now, lets see some real code. To focus on our topic, I am just concentrating on the plumbing part.

Step-1

Develop a SLSB called StockWS. Annotate it with @Webservice to make it a JSR 109 webservice. See the code snippet below.

Deploy the service to the application server (Project GlassFish).

Step-2

Use BPEL designer to design a BPEL app. I am not explaining how exactly this needs to be done. Take a look at this article (its a bit too long) or read gopalan for details.

Basically, you need to import the WSDL of the SLSB webservice to the BPEL application and design the BPEL process using the BPEL designer in the netbeans enterprise pack. At the end of everything, the BPEL app will expose a service as shown below in the WSDL snippet.

Step-3>

Create a composite application project, add the BPEL app to the composite application project and deploy it. These are few clicks in the netbeans enterprise pack.

Step-4
Write the MDB. Import the WSDL of the BPEL application to the MDB as webservice client. The MDB snippet below explains everything.

Step-5
The last part of the story. Normally, the above JAXWS code in MDB will try to access a service using HTTP transport. Here we want the MDB to access the BPEL app in the same JVM. So, tweak the mdb deployment descriptor to route it to JBI.

Java EE Service Engine is the glue code that is part of Sun Java System Application Server. One end of the service engine exposes JBI based contracts and is deployed into JBI runtime. The other end interact with appserver's containers, especially the webservices runtime. When the JAXWS client code executes, using a private SPI, it snatches the soap message from the JAXWS client runtime and normalizes and sends it to JBI NMR. Similarly, when the JBI NMR invokes Java EE Service Engine with an incoming JBI request, Java EE Service Engine will pass the soap message to JAXWS server runtime and get the jsr 109 webservice (slsb) invoked. With the new re-architectured JAXWS runtime (Project Tango) we should be able to change all these into a more pluggable architecture.

For most simple applications, it is still okay. Most JMS providers give some sort of JNDI capability so that an application can lookup javax.jms.QueueConnectionFactory and javax.jms.Queue objects and send a message.

The trouble comes, when you need to write a true J2EE application, that use container managed transaction and authentication management, that need to be performant (think of connection pools), need inbound communication with MDBs etc. Then add distributed (XA) transactions to the mix. It gets very complicated.

More than an year after we started http://genericjmsra.dev.java.net project, I did a bit of googling about Generic RA for JMS. It seems, many people are able to use different JMS providers with different application servers using this resource adapter.

The resource adapter is now bundled with two application servers. Sun Java System Application server (Project GlassFish) and Borland Application Server. It is also known to work with Websphere Application Server. These links (1 and 2) indicate that someone has got it working with Geronimo also. Hopefully, more application servers will get into the list sooner or later.

Look like jencks folks have got it working with their container. Since jenks is very close to Spring, it also can connect to any JMS provider using the resource adapter.

It has also been found working with the many JMS providers. WebsphereMQ, Tibco EMS, OpenJMS, ActiveMQ, Sun Java System Message Queue and Seebeyond Message Queue are found to be working with one or more application servers. Hopefully someone will try others (FioranoMQ, SonicMQ, JbossMQ, SwiftMQ etc) also soon.

Siva recently gave a community corner talk titled the availabilty features in the genericjmsra at JavaOne 2006. Those slides will be useful for folks who want to use the adapter in production. The other good resource is the user guide.

Please checkout the slides of Project GlassFish talk. It contains an overview of Java EE 5 and Sun Java System Application Server specific features.

If you attend the event, visit Sun stall to see more demos from the glassfish team.

Lets come to the topic.

I got many e-mails recently, asking me details about a feature called on-demand initialization (or lazy initialization) in Sun Java System Application Server or GlassFish ... and I thought I could use this blog post to explain what it is all about.

What does this feature accomplish?

After this feature is implemented, the actual startup time for Application Server became nearly half of what it was in its previous version. The memory footprint has also reduced a lot. See Peter's blog about his experience

Why was it taking longer to startup earlier?

In my mind, Java EE is a mature and really capable technology for enterprise applications. As an Application Server compliant to Java EE, GlassFish supports many technologies or services specified in the Java EE specification. Lets try to name a few of them... Servlet, JSP, Webservices, EJB, JMS, Connectors, JDBC, JMX, JSF....

Application server, on startup, initialize containers/components that provide these services eagerly and this takes a long time AND lot of memory.

So, what does this new feature do?

Most of the developers are not going to use all the services in the Application Server. I am not talking about those who develop really complex EE applications all the time, but about developers who download the Application Server and just try out a few samples, or folks who learn one or two technologies at a time or even folks who work on small projects that use only some of the technologies from the Java EE stack. This group of developers would not like to incur the overhead of startup-time/memory for all the services they would not use.

Hence now, the Sun Java System Application Server does not start all the containers during startup. In fact when you install Application Server and start it afresh, there will be only a handful essential services initialized, namely a local JNDI provider, JMX connector (for administration).

Now, how would a developer configure the application server to use the
other services?

That is the best part of the feature. User doesn't need to do anything special to initialize the other services that are not started by default. The user uses the application server as he normally would. As and when a service is used, the Application Server will transparently start it.

Here are some details about the implementation.

Mainly there are 3 "entry points" to the Application Server. The entry points are basic functionality in the Application Server that will be used while accessing a user-facing service in Application Server.

1. JNDI Lookup.

Many services and resources in Application Server are looked up using a JNDI name. Examples?
Looking up an EJB, a JDBC resource, a JMS connection factory...

2. Server Ports.

Whenever you do something with the Application Server, a server port is accessed. It can be when you try to see the index.html of the webserver , or connect to the JMS provider from a standalone application, or it can be when you try to lookup a JDBC resource from server's JNDI provider from a standalone Java application.

3. Application Loader.

During the loading of an application,the application Server will read the deployment descriptor of the application for house keeping operations and initialization of the application.

Whenever a user-request reach any of these entry points,the Application Server will identify the type of incoming request and start all relevent services (or service groups) that are required. For example, if the user is accessing http port and if webserver is not started, then the webserver is started lazily/transparently. Or if user is accessing JMS port and JMS provider is not started, Application Server will start the JMS provider. When an ear file containing a EJB is loaded, and ORB is not yet started, Application Server will start ORB.

If you are interested in this feature, the code is available for you to see at GlassFish FishEye page.

If you like to download the latest MSbuild of glassfish, see the link here

For any questions/comments, please use the comments section below or post them at the GlassFish developer's mailing list dev@glassfish.dev.java.net