Claudio Miranda's Blog

Heap dump on Linux 64 bits

I am working for a customer, doing some performance review for their java application and tuning glassfish as well.

They use linux 64 bits (kernel 2.6.18 SMP), glassfish v2 u2 and JDK 5 u12. At some point I needed to generate a heap dump to better understand the object allocation and if possible some hints as to where to look for optimization for the application.

Now the big problem, I could not generate a heap dump, not with the current tools available at that time. I have tried

• -XX:+HeapDumpOnCtrlBreak and kill -3
  
• jmap -heap:format=b
• gcore utility
  

All of them generated an sun.jvm.hotspot.debugger.UnmappedAddressException, except gcore who just killed the process. See the stacktrace

# /usr/local/jdk/jdk1.6.0_07/bin/jmap -J-d64 -F -dump:file=java_dump_10791.hprof  10791
Attaching to process ID 10791, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 10.0-b23
Dumping heap to java_dump_10791.hprof ...
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.tools.jmap.JMap.runTool(JMap.java:178)
        at sun.tools.jmap.JMap.main(JMap.java:110)
Caused by: sun.jvm.hotspot.debugger.UnmappedAddressException
        at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
        at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
        at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
        at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readOopHandle(LinuxDebuggerLocal.java:431)
        at sun.jvm.hotspot.debugger.linux.LinuxAddress.getOopHandleAt(LinuxAddress.java:115)
        at sun.jvm.hotspot.oops.Oop.getKlassForOopHandle(Oop.java:222)
        at sun.jvm.hotspot.oops.ObjectHeap.newOop(ObjectHeap.java:348)
        at sun.jvm.hotspot.utilities.HashtableEntry.literal(HashtableEntry.java:53)
        at sun.jvm.hotspot.memory.SymbolTable.symbolsDo(SymbolTable.java:106)
        at sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeSymbols(HeapHprofBinWriter.java:830)
        at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:396)
        at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:56)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
        at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77)

UPDATE: The stacktrace displayed above, shows a dump request from a JDK 6u7 to a target VM 6 u7.
UPDATE: Previously, I had issued jmap from JDK 5 u12 to a target VM 5 u12. The stacktrace is the same.
UPDATE: The target VM is not using -Xrs option.
UPDATE: The VM process owner is the same who issued the jmap command, root.

UPDATE: Alan Bateman clarified a bit about the -F option, "The -F options causes the tool to attach to the target VM in a non-cooperative way and so may observe the heap in an inconsistent state. In other words, there is no guarantee that you will get a good heap dump when you using the -F option" read more on the comment section of my portuguese blog (the blog post is the same as posted here).

I tried JDK 6 u7, with the same error.

Later on, I found a bug "Throws UnmappedAddressException while reading address from core file in shared area."

It is fixed for JDK 6 u10 RC (changelog), so I really shoud give it a try. Don't forget the -Xshare:off option.

And it worked as it should be, after heap dump was generated the JVM works as normal.

So if somebody wants to generate heap dump on linux 64 bits, try to use JDK 6 u10 RC.

Looks like I need to say "Thank You" to Tony Printezis, as at the end of heap dump generation, there is a hint about the author's code.

"Finding object size using Printezis bits and skipping over..."

Comments

This work is licensed under a Creative Commons License.