Skip to main content

Heap dump on Linux 64 bits

Posted by claudio on October 1, 2008 at 8:52 PM PDT

I am working for a customer, doing some performance review for their java application and tuning glassfish as well.

They use linux 64 bits (kernel 2.6.18 SMP), glassfish v2 u2 and JDK 5 u12. At some point I needed to generate a heap dump to better understand the object allocation and if possible some hints as to where to look for optimization for the application.

Now the big problem, I could not generate a heap dump, not with the current tools available at that time. I have tried

  • -XX:+HeapDumpOnCtrlBreak and kill -3
  • jmap -heap:format=b
  • gcore utility

All of them generated an sun.jvm.hotspot.debugger.UnmappedAddressException, except gcore who just killed the process. See the stacktrace

# /usr/local/jdk/jdk1.6.0_07/bin/jmap -J-d64 -F -dump:file=java_dump_10791.hprof  10791
Attaching to process ID 10791, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 10.0-b23
Dumping heap to java_dump_10791.hprof ...
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.tools.jmap.JMap.runTool(JMap.java:178)
        at sun.tools.jmap.JMap.main(JMap.java:110)
Caused by: sun.jvm.hotspot.debugger.UnmappedAddressException
        at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
        at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
        at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471)
        at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readOopHandle(LinuxDebuggerLocal.java:431)
        at sun.jvm.hotspot.debugger.linux.LinuxAddress.getOopHandleAt(LinuxAddress.java:115)
        at sun.jvm.hotspot.oops.Oop.getKlassForOopHandle(Oop.java:222)
        at sun.jvm.hotspot.oops.ObjectHeap.newOop(ObjectHeap.java:348)
        at sun.jvm.hotspot.utilities.HashtableEntry.literal(HashtableEntry.java:53)
        at sun.jvm.hotspot.memory.SymbolTable.symbolsDo(SymbolTable.java:106)
        at sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeSymbols(HeapHprofBinWriter.java:830)
        at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:396)
        at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:56)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
        at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77)


UPDATE
: The stacktrace displayed above, shows a dump request from a JDK 6u7 to a target VM 6 u7.
UPDATE: Previously, I had issued jmap from JDK 5 u12 to a target VM 5 u12. The stacktrace is the same.
UPDATE: The target VM is not using -Xrs option.
UPDATE: The VM process owner is the same who issued the jmap command, root.

UPDATE: Alan Bateman clarified a bit about the -F option, "The -F options causes the tool to attach to the target VM in a
non-cooperative way and so may observe the heap in an inconsistent
state. In other words, there is no guarantee that you will get a good
heap dump when you using the -F option
" read more on the comment section of my portuguese blog (the blog post is the same as posted here).

I tried JDK 6 u7, with the same error.

Later on, I found a bug "Throws UnmappedAddressException while reading address from core file in shared area."

It is fixed for JDK 6 u10 RC (changelog), so I really shoud give it a try. Don't forget the -Xshare:off option.

And it worked as it should be, after heap dump was generated the JVM works as normal.

So if somebody wants to generate heap dump on linux 64 bits, try to use JDK 6 u10 RC.

Looks like I need to say "Thank You" to Tony Printezis, as at the end of heap dump generation, there is a hint about the author's code.

"Finding object size using Printezis bits and skipping over..."

Related Topics >>

Comments

Forgot to mention that we were trying both with and without -Xshare:off.

Hi, sorry for not answering (I didn't get any notifications of your response). The problem persists, but it doesn't happen always. E.g. I've never experienced this problem when doing a dump from a JBoss with no application deployed. However after deploying our EAR, the dump fails. The same problem has appeared to my colleague, who was trying to a dump on the same machine but with JDK5: $ /opt/java/jdk5/bin/java -version java version "1.5.0_19" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_19-b02) Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_19-b02, mixed mode) (of course the jmap and the JBoss were run from the very same JAVA_HOME)

Hi @grzegorz: Did you try to run the target JVM with -Xshare:off option ?

I keep getting the same error even after updating from JDK6u7 to JDK6u14: $ /opt/java/jdk16014/bin/jmap -heap:format=b 16227 Attaching to process ID 16227, please wait... Debugger attached successfully. Server compiler detected. JVM version is 14.0-b16 Dumping heap to heap.bin ... Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jmap.JMap.runTool(JMap.java:179) at sun.tools.jmap.JMap.main(JMap.java:110) Caused by: sun.jvm.hotspot.debugger.UnmappedAddressException at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208) at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63) at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:217) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:482) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readOopHandle(LinuxDebuggerLocal.java:436) at sun.jvm.hotspot.debugger.linux.LinuxAddress.getOopHandleAt(LinuxAddress.java:120) at sun.jvm.hotspot.types.basic.BasicField.getOopHandle(BasicField.java:175) at sun.jvm.hotspot.types.basic.BasicOopField.getValue(BasicOopField.java:53) at sun.jvm.hotspot.utilities.HashtableEntry.literal(HashtableEntry.java:53) at sun.jvm.hotspot.memory.SymbolTable.symbolsDo(SymbolTable.java:106) at sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeSymbols(HeapHprofBinWriter.java:838) at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:396) at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:56) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77) ... 6 more $ /opt/java/jdk16014/bin/java -version java version "1.6.0_14" Java(TM) SE Runtime Environment (build 1.6.0_14-b08) Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)