Kohsuke Kawaguchi's Blog

Debugging java, at native level

This is the record of how I debugged JVM at native code level. This is neither a complete guide nor do I work in the JavaVM team. This is just in the hope of helping others in the same boat.

There has been a long standing Hudson bug where Maven launched by Hudson in a certain way hangs on Windows. Sometimes when you are looking at a bug, you see that the bug is deep down the software stack, and this bug was one of those. If you start with a debugger support, the problem will be gone. Or if you attach jconsole, you don't see any Java level dead lock. The main Maven execution thread just blocks on gethostbyaddr (or gethostbyname, I forgot), and you have no idea how that could be.

I really couldn't figure out what's going on at Java debugger level, so I had to resort to using the native debugger and dive into the JDK.

1. First, you need Visual Studio 2008 to do this. If you haven't bought a license, you can download the express version for free, and that would just do fine.
2. Next, set up the symbol server so that you can see Windows API by symbol names, not just by addresses. See this MSDN article for more details.
3. Get the JDK source code. I have access to internal JDK archive, so I can access the source code of any build of JDK, but on outside I can only find this. This means you should also download the corresponding JDK binary bundle and run it, to avoid line mismatch.
4. When you attach a debugger process to JVM, it should automatically load the symbol database for JVM, because JDK ships with "*.pdb" files. See your jre/bin/*.pdb files. The symbol files contain function names and their offsets in the DLL files, as well as source file names where that came from.
5. When you navigate around the call stack of threads, Visual Studio will ask you where the source files are. Chances are you've extracted source archive in a different directory than the JDK official build, you need to let Visual Studio know where they are in your HDD. But if you do that, you get to see the source code, not just the disassembly. Yay!!

Now, the main thing I couldn't figure out is how to relate code produced by JIT to their corresponding Java method names. Because of this, I couldn't figure out which Win32 thread corresponds to which Java thread, and where they are executing. If anyone knows how to do this, please share it with us.

But still, I think this is an useful way to get into the gory details of how JDK works. It'll be also useful if you are debugging JNI part of your app, because now you can see who's calling your method or how the methods on JNIEnv are behaving.

Comments

• You should also be able to do this with the Windbg tool (free) instead of requiring Visual Studio. http://www.microsoft.com/whdc/devtools/debugging/default.mspx

  Posted by: jchristi on June 05, 2008 at 09:02 PM

• Visual Studio Express is also free, though. Is there any reason one should prefer WinDbg, which IIRC is a kernel debugger?

  Posted by: kohsuke on June 06, 2008 at 06:53 AM

• Did you find the bug? What was that?

  Posted by: felipeal on June 06, 2008 at 11:40 AM

• No, I didn't. In the end I just chickened out — I throw away a part of the code and did it in an entirely different way, in the hope that the problem will be gone.

  Posted by: kohsuke on June 06, 2008 at 03:32 PM

This work is licensed under a Creative Commons License.