Xiaobin Lu's Blog

Perception == Reality

Perception vs. Reality: How big is that process?

One of the issues we have run into our work on the Java performance team is the difference between the physical memory size of a process and the perceived size of that process. "Perceived" footprint is the number reported by the operating system (by tools such as "ps" and "top" on Unix/Linux, and "Task Manager" on Windows) that a process takes up in RAM. In reality, the size a process takes up in memory consists of executable code, global data, stack and heap etc. However, to the end-user running one of these process-viewing tools, the size of a process is much different since sometimes these tools include some memory that is not actually allocated in RAM at all. The gap between perceived and real footprint is significant when memory mapping is used, particularly on Solaris & Linux. For those who don't know what memory mapping is, memory mapping is a technique which treats file I/O, such as "read()" and "write()" system calls, as routine memory accesses. For example, you could simply dereference a pointer obtained from memory mapping the file. Memory mapping not only greatly simplifies programs dealing with file I/O, but also allows multiple processes to share the same underlying file data. However, the downside is these process-viewing tools such as "ps" and "top" will include the memory mapped region size even you don't touch the file at all.

How does it matter to Java?

Java historically memory maps all of its jar files in the JDK into the memory address space on Unix platform. One of these jar files is "rt.jar", which contains all class files in JRE and is about 40 Megabytes in size. Obviously that makes Java looks a lot bigger than it actually is.

Why do we care about perception?

The main reason we care about perceived footprint is that, in the eyes of end-users, Java processes look much larger than they actually are, which makes Java look like a resource hog. End users will not know or care about distinctions between perceived and real footprint, nor the processes for measuring footprint with memory mapped files; they will just see a Java process taking up what looks like a lot of memory. Why not give these users information that is closer to reality and avoid the speculation and misinformation that arises from the perceived footprint numbers?

What are we doing about it?

As of build 45 in Mustang, we have fixed this problem and made perception align better with reality. Instead of memory mapping jar files, we now read their contents from the files just as we have always done on Windows. We still memory map a very small portion of the file, called the "central directory", since that section contains information which is used frequently and can benefit from the sharing and mapping capabilities of mmap. But otherwise, the contents are simply read in as needed with normal file loading operations.

What about the sharing benefits of mmap?

Actually, the contents of rt.jar are not shared very much in practice. For one thing, we introduced class data sharing in J2SE 5.0, which loads most of the core class content from a different file entirely and ignores most of the contents in rt.jar. But even without class data sharing, we would typically only use the mmap'd data once per process, at startup, and this sharing is not worth maintaining long-lasting data structures (or long-lasting footprint penalties). By doing what I just described, we effectively bring the perceived footprint down by around 55% on Solaris and 25% down on Linux platform measured with a set of internal benchmarks.

But Wait, There's More!

The main reason we pursued this project was to combat the perceived footprint issue; and by the numbers quoted above, we think that goal was achieved. But another huge benefit fell out of this work; we're actually noticing a 11% decrease in real footprint on Solaris and Linux. It turns out that the process of rewriting this code and sharing the implementation with Windows (which always used read() instead of mmap()) made for many more efficiencies in the implementation and ended up in a substatial drop in real footprint. Another benefit is simply more efficient use of the memory resources at our disposal. This change reduces the risk of address space exhaustion (resulting in java.lang.OutOfMemory exceptions) on 32-bit platforms when opening very large jar files. Also, we no longer have the problem of mmap'd jar files competing with other memory demands such as regular java heap and native memory.

Perception == Reality

25 - 55% drop in perceived process memory size? 11% drop in real memory footprint? No matter how you perceive it, the benefits are real.

Comments

• This is totally cool. Memory usage is a major complaint by people using Java, and when I try to tell them about how top lies about memory usage, they say they believe me, but I don't think they do. Hopefully now it won't be as big of an issue.

  Posted by: keithkml on August 09, 2005 at 11:04 AM

• Happiness! I've been regularly surprised by the number of people both inside and outide of Sun who get confused by the misleading Solaris java size numbers from top and ps. The new numbers give a much more realistic view of how much physical memory is actually getting used!
           
           
           
           
  - Graham
  

  Posted by: kgh on August 09, 2005 at 01:09 PM

• I hope you meant perception.equals(reality). I hope the world around me isn't stored in the same location as my perception of it! :-)

  Posted by: heaththegreat on August 10, 2005 at 07:38 AM

• Thanks for this info. We have always wondered why top showed the java process consuming 250+MB of RAM when the max heap size was still set at the default.

  Posted by: troydugger on August 10, 2005 at 07:57 AM

• If you reduce something by 120% doesn't it become -20% of what it initially was? I guess what you meant: down by 55%.

  Posted by: karakfa on August 10, 2005 at 08:04 AM

• This is great. A question, however:

  Would not 120% drop result in a negative memory size? Is it a 20% drop?
  

  Posted by: jtr on August 10, 2005 at 08:48 AM

• Very well explained ! Thanks for for the great blog Xiaobin.. and keep up the good work !

  Posted by: gmanwani on August 10, 2005 at 08:49 AM

• Thanks for the comments. I just have a slight clarification for the data in the blog. The "120%" should actually be "55%", 30% should be 25%. The real footprint actually dropped 11%.

  Somehow I interpreted the number in the other way. Sorry for the confusion.

  Posted by: xiaobinlu on August 10, 2005 at 10:20 AM

• Xiaobin, you rule! Thanks for the good work and informative post. I have often been surprised by top, and now I understand why.
  

  Posted by: erickson on August 10, 2005 at 11:27 AM

• nice, still i am using winxp and comparing the diferences of a java vs a native app they still look like major memory hogs plus one has to use special tool like process explorer from sysinternal to get pass the process identification limits (javaw) it would be nice if you fixed that problem.

  Posted by: danielmd on August 10, 2005 at 01:22 PM

• For the last comment from "danielmd", I would suggest you to file a bug at http://bugs.sun.com/bugdatabase/index.jsp. Thanks.

  Posted by: xiaobinlu on August 10, 2005 at 02:15 PM

• I guess these 25% to 55% numbers are relative to earlier JDK 1.6 builds? I just did a quick check of "top" on linux and found that a "hello, world" program with JDK 1.5: 9104/9100 (SIZE/RSS) while JDK 1.6B46 is 9092/9088. With ps -auxw, I see JDK 1.6: 214780/9860 JDK 1.5: 262600/9096

  That's a decent decrease in reported virtual memory size. I see JDK 1.5 as having a smaller resident size, so I guess the 11% number is against previous JDK 1.6 builds?
  

  Posted by: jtr on August 10, 2005 at 03:26 PM

• I should made it clear that the number I got is relative to b44 of JDK 1.6.0. So the answer to the last comment is "yes", the 11% number is against previous JDK 1.6 builds.

  Posted by: xiaobinlu on August 10, 2005 at 03:35 PM

• Thanks for this article. I've been trying to track down a possible memory leak / or certainly strange memory behavior for over a month now, and this article has shed some light on a possible explanation.

  I've used jconsole to monitor memory used by the application, both in heap space and non-heap memory, and memory use is stable and GC works fine. I've also used jprobe and didn't find any lingering objects. All JDBC objects closed in a finally clause, no other jni,etc...

  The memory reported by java is a small fraction of the allocated space: Output follows,

  Yet when I observe memory consumption reported by top, it seems to grow, stabilize, shrink, then grow some more, and the memory use does not correlate to system load or any other factor.

  This environment is running Tomcat 4.1 on linux with jdk1.5.0_04 running with min/max stack size of 128m.

  Here is a graph of the memory usage over a 2 day period.
  Memory graph
  Could someone please tell me if this seems like an actual memory leak, or typical of how memory use is reported to linux. What concerns me is the SIZE and RSS fields growing over 430 and 330mb respectively, when if you add 128mb heap + another 100mb for native mem + 40 for rt.jar (if not included in native space) and we're still well under 300mb.

  
  Thanks,
  David

  Memory type=Non-heap memory Memory usage=init = 163840(160K) used = 6326848(6178K) committed = 6356992(6208K) max = 33554432(32768K)
  Memory type=Heap memory Memory usage=init = 8323072(8128K) used = 9360(9K) committed = 8323072(8128K) max = 8323072(8128K)
  Memory type=Heap memory Memory usage=init = 983040(960K) used = 0(0K) committed = 983040(960K) max = 983040(960K)
  Memory type=Heap memory Memory usage=init = 123928576(121024K) used = 15771544(15401K) committed = 123928576(121024K) max = 123928576(121024K)
  Memory type=Non-heap memory Memory usage=init = 8388608(8192K) used = 16359648(15976K) committed = 16515072(16128K) max = 67108864(65536K)

  Posted by: weathermaps on September 03, 2005 at 09:58 PM

• Per last comment, I think the more appropriate place to ask such question is the forum in java.net (http://forums.java.net/jive/index.jspa). But anyways, have you tried to use "OptimizeIt" or "Netbeans profiler" (or JProfiler)?
  That may help you to find out the memory leaks.

  Posted by: xiaobinlu on September 05, 2005 at 12:10 PM

• Thanks for your suggestion. I've tried both JProbe and JProfiler and neither showed any evidence of memory leaks(objects piling up). Looking at the memory usage reports from java, both native and non-native memory are reporting using only a fraction of their allocated space. Yet, the memory usage as reported by top is significantly larger and growing (see graph) and it seems as if some sort of memory deallocation is going on(from the periodic dips) which is independant from the Heap GC.
  I'm not sure if the memory use as reported by top is a sign of a memory leak, or just a side-effect of the way java provides memory usage information. In other words, in your experience from java 'over-reporting' its memory usage, is it simply a fixed amount, or can it increase with time? I just want to make sure there is a legitimate memory leak before I spend more time on this.

  I wasn't sure this was the appropriate forum, but I've posted several messages at forum.java.sun.com with little success.
  Thanks
  you can also reach me at weathermaps@gmail.com

  Posted by: weathermaps on September 05, 2005 at 12:53 PM

• The memory size shown as top should not incease as application runs. If it does, I believe it is a memory leak. This work I've done just try to reflect what it is really going on internally. I have no intention to solve any memory leaks.

  Posted by: xiaobinlu on September 05, 2005 at 01:28 PM

This work is licensed under a Creative Commons License.