Why, then, have I recently switched to using the NetBeans profiler? Yes, it's free and powerful -- but that hasn't gotten me to switch away from vi and javac. And I'm a long-time user of OptimizeIt when I need to profile Java applications on Windows machines. Granted, my use of OptimizeIt is limited to Windows machines; for profiling on Sparc and Linux, I much prefer Sun's Studio 11 collector and analyzer; the fact that those tools are written in C means they can give me insight into the JVM that no Java-based profiler can. But still: what overcame my usual inertia and got me to move to the NetBeans profiler?

Quite simply, it works when other tools don't. For quite some time, the glassfish performance team has faced a regression in deployment times that manifests itself only on Windows. Months of analysis with OptimizeIt failed to make any progress. Fifteen minutes with the NetBeans profiler, and we had a fix.

Well, twenty minutes. One caveat about using the NetBeans profiler with glassfish: glassfish doesn't like the default installation path of Netbeans (or any path that contains a space in it). When it comes time to configure glassfish, if the path to the Netbeans profiler has a space, you'll face some difficulty. So my first recommendation: grab the Netbeans platform and profiler, and install it in a non-standard location (e.g., C:\NETBEANS).

Once it's installed, make sure to add the Netbeans profiler directory to your PATH variable (e.g. C:\NETBEANS\profiler\lib\deployed\jdk15\windows). Then fire up Netbeans, and click on the Profile menu and select the "Attach profiler" menu item. This displays a window, and the first thing you should notice in that window is that it contains an "Attach Wizard" button. That's right: to configure glassfish (and many other applications) to work with the Netbeans profiler, all you need to is proceed through a simple GUI. No dealing with difficult shell scripts to start your appserver [Hey, these productivity features might be worth it after all.]

I won't go into the details of the attach GUI, other than to point out that because glassfish is not yet released, Netbeans doesn't have a choice for that: you must select the Sun Java Systems Application Server 8.1. Because that is configured the same way as glassfish, you're all set. [If by chance you installed NetBeans in the default location, pay attention to the configuration info that the wizard tells you it added to the appserver's domain.xml file: you must edit the paths in that file to conform to that really intuitive Windows file naming convention that uses lots of UPPER~ names.]

Ease of use is a great thing, but that's not good enough for me. What did the tool show us that convinced me to switch? We started the appserver in profiling mode. After it started up, we reset the collected results in the profiling window (hmm, another cool feature), deployed our application, and took the following snapshot:
Like all Java-based profiers, the Netbeans profiler is somewhat confused about methods that block on I/O, so the poll method looks to be the most time-consuming. However, we know that can be discounted: the method has blocked in the kernel so isn't taking any CPU time. But the ZipFile.access$700 method is certainly suspicious. Indeed, repeating our experiment on the Sun Java Systems Application Server 8.2 (from which we were measuring the regression) makes it that much clearer: in the 8.2 profile, the access$700 method is called only some 40K times and requires only about 4999 ms of CPU time.

Looking at the backtraces of that method in glassfish, we see that it is called via two paths:
One path is clearly the compiler and is called 47K times; the other path turns out (when you expand the nextElement$1 node) to be from the creation of the EJB classloader associated with the application. In 8.2, however, the access$700 method is called only from the compiler; it is never called from the EJB classloader. A few minutes of detective work and we found that a change in the code creating the EJBClassloader meant that system jar files were incorrectly being added its search path, and reading those jar files led to all the new calls to the access$700 method.

We were searching for a regression, but note how easy this still would have been if our goal had simply been to make deployment faster: we would have immediately started to look for ways to call the access$700 method less often and been led to the same classloader code. With minimal effort, we know that to make deployment faster, we need to optimize the way in which the EJB classloader is created. [Strictly speaking, the conclusion we draw is that we are even better off making the compiler optimize the way in which it calls the access$700 method. But the compiler comes to us from another team; we can (and have) asked that team to optimize the code, but it's not something we can affect directly.]

Our OptimizeIt profiles never showed us this issue. They did point to the access$700 method as a possible hotspot (one of about eight!), but the call stack shown by OptimizeIt for that method shows only the compiler path: it was impossible to detect from the OptimizeIt profile that the access$700 method was ever called by anything but the javac compiler. A lengthy investigation (before the more accurate information from NetBeans was available) led us to prove that the javac invocation was identical and taking an identical amount of time; an unfortunate blind alley. And then the OptimizeIt profile led us completely astray to examine the other hotspots in the profile.

Had OptimizeIt solved my problem in the beginning, I might never have gotten around to trying the NetBeans profiler, despite entreaties from my friends and colleagues at NetBeans. Since Netbeans actually solved problems for me that nothing else could, I'm now its biggest fan. Now if it only had a vi keybinding mode for its editor...