Kelly O'Hair's Blog

Needing a New Heap Dump Format

I spent quite a bit of time working on HPROF in the JDK 5 (Tiger) release, converting it from JVMPI to JVM TI. But I also spent a great deal of time making sure the new code created output files that matched the HPROF file formats byte for byte. This was not an easy task, not only were these file formats poorly specified, HPROF itself was considered "experimental", built on an "experimental" JVMPI which influenced the file format design, and we were severely limited as to the available tests to verify the file formats. In addition, we had no idea who used the file formats, or what we could safely change in them. Usually, nothing was changed in these files because of this.

Turned out with a few minor problems, most tools just worked with the files created by the JDK 5 HPROF. But it was obvious that this was less than an ideal situation, for everyone. We needed a better specification, and the file format needed some updating. At the same time it was obvious that changing these formats could create some severe compatibility problems for some vendors.

During the JDK 6 Mustang development, some industrious co-workers (Alan and Sundar) started creating heap dumps from running VM applications, without HPROF, but using the historical HPROF binary format (it was decided that creating a new format at the time wasn't a good idea). Then also in JDK 6 the original HAT utility that reads HPROF binary files, was transformed into jhat and added to the JDK. Now we had multiple ways to create HPROF binary files, but we had added a utility that could at least be used in tests to verify the format had the correct syntax. But wait, there's more!

In Sundar's Blog you can see that he has taken it even further, merging some scripting work he has done with the functionality of jhat. If you haven't looked at Sundar's work, or tried what is available in Mustang, you should give it a try.

And all this is based on that ancient ill-specified HPROF binary file format. It was obvious that these heap snapshot files had some un-tapped value.

So I've started a small JDK project to define a new heap dump format at:

https://heap-snapshot.dev.java.net/

Hopefully we can come up with a format that is as good as the old HPROF binary format and ideally much better, with a formal specification.

If you are interested, please join the project and let's have some discussions about this. If we can standardize a format it should serve as a solid basis for more tools and support in helping analyze and diagnose heap related problems in java applications.

Comments

This work is licensed under a Creative Commons License.