Eamonn McManus's Blog

How much does it cost to monitor an app with jconsole?

Recently I've seen several people ask what the cost of enabling JMX monitoring on an application is. If I run with -Dcom.sun.management.jmxremote and connect jconsole, how much will that affect the performance of my app? Here are the results of some highly unscientific experiments.

Here's the quick answer for people who look at the answers in the back of the book. Running with -Dcom.sun.management.jmxremote has no appreciable impact. Connecting jconsole has an impact of 3--4%.

The answer to this question is likely to depend very much on the nature of the app and what else is going on in the machine where it is running. My measurements here are on a toy application which computes digits of the mathematical constant e using BigInteger arithmetic. This isn't intended to be an illustration of appropriate mathematical techniques. It's just a simple way to create an app that does a lot of computation and a lot of allocation of large short-lived objects. So my measurements here are likely to be interesting for apps like that, and probably not wildly irrelevant for other sorts of apps.

Here for reference is the method that the app spends most of its time in:

    private static void computeE(int digits) {
        BigInteger one = BigInteger.TEN.pow(digits);
        BigInteger e = BigInteger.ZERO;
        BigInteger invfact = one;
        BigInteger n = BigInteger.ONE;
        while (invfact.compareTo(BigInteger.ZERO) > 0) {
            e = e.add(invfact);
            invfact = invfact.divide(n);
            n = n.add(BigInteger.ONE);
        }
        System.out.println(e);
    }

Note by the way that the last few digits computed by this method are inaccurate due to truncation of the division results.

I ran this method with the digits parameter equal to 30000 on an elderly SPARC machine. I was logged in to the machine in an X Windows session too, so there was some background activity. I ran each measurement two or three times for each configuration and took the smallest time. (I did say this was unscientific. What are these repeatable test conditions and standard deviations of which you speak?)

I created one thread per CPU to make sure that the machine was CPU-bound. Otherwise we might see zero overhead for enabling monitoring because it might be using an otherwise-idle CPU.

Startup time is slightly greater when you enable monitoring because it needs to do some extra work such as creating an RMI connector. I assumed that the figure of interest, though, is the cost while the app is running. So I measured the time between the creation of the computeE threads and their completion.

I used two Java platforms for the measurements: Sun's JDK 1.5.0_07 and a Mustang snapshot (1.6.0-rc-b92). The time for the computation was 37% better on Mustang, presumably because of improvements to the JIT compiler and/or heap management. Also the time is better still if you run with -server. But what we're interested is the relative difference between monitoring and not monitoring.

The first measurement was of the application running without any special command-line parameters. So monitoring was not enabled. This is the baseline for the other measurements.

Then I ran the application with -Dcom.sun.management.jmxremote, which has two effects:

• It creates the Platform MBean Server and populates it with the standard set of MBeans containing the JVM's instrumentation.
• It creates and starts an RMI Connector Server and puts its address where jconsole can find it.

So with this option we could connect jconsole from the local machine. But I didn't connect jconsole for this measurement. The effect on performance was less than 1% in this case, and in fact any observed difference was less than the margin of error. So there was no significant difference in the time to do the computation. In other words, so long as you don't connect jconsole, enabling monitoring should have no effect on your app's performance.

I got the same result when I ran with
-Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.remote.ssl=false
so allowing jconsole to connect remotely but without security also has no effect on performance. Note that we strongly discourage running apps this way in production environments.

Running with
-Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.password.file=jmxremote.password
(meaning there is security including the creation of an SSL socket) also has an impact of less than 1%.

Then I ran the insecure options again but this time connected Tiger's jconsole from another machine. Here are the figures I got:

So here there's a fairly big impact. Some of this is due to jconsole polling the various JVM MBeans so it can graph the changes in various values like heap size and number of threads over time. There is no way to stop jconsole from doing that, even if you're not interested in that information, but what you can do is change the polling interval. If you change the interval from the default 4 seconds to 1000 seconds, say, then you won't see much impact from polling. (The interval also applies to the graphing of your own attributes, though, so if you want that then you'll have to pay for the JVM polling.)

Running Tiger's jconsole with -interval=1000 changes the figures to this:

Jconsole isn't polling, but there's still some overhead, presumably due to background activity from RMI and/or to increased memory usage because of the open connection.

Mustang's jconsole, besides being generally nicer, has an optimization that means it obtains the various JVM attributes much more efficiently, so the polling impact is less. Even if you can't migrate your apps to Mustang right now, you might want to download the Mustang JDK just so you can run its jconsole!

Here are the figures running the app with the insecure options and connecting with Mustang's jconsole from another machine:

So you get better behaviour from jconsole by default here. And of course you can still use the -interval=1000 trick, to get this:

Finally, I tried a few configurations with security enabled (-Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.authenticate=true, which are the default settings) and did not see any signficant difference. Even with jconsole polling every four seconds, there was no observable extra overhead due to SSL.

Reminder

These results are not conclusive! They're merely indicative of one set of measurements on one type of app. The impact could be much greater if the addition of the JMX connection pushed the app into a different mode of operation. For example, the additional memory usage could be enough to change the behaviour of the garbage collector significantly and probably negatively. But you would have to be operating fairly close to the edge of such a possible change for that to happen, meaning it could happen even without the JMX connection if for example the app needed to handle a few more objects.

Comments

• Great post! I have been wanting to experiment with JMX more, but never seem to carve out the time. I hadn't looked at jConsole before, but gave it a try after reading your post and it sure is easy to enable. The amount of information that you get is fantastic and could really help with debugging--not to mention peaking in on those production servers once and a while.

  --Larry

  Posted by: ldrothny on July 21, 2006 at 08:46 PM

• Hi Eamonn,
  We recently released JXInsight 4.2 which includes 'metrics' and 'jvminsight' extensions for JMX. The JXInsight agent performs all its metric sampling within the JVM and thus its impact should be much lowering than the large number of RPC's calls being generated by JConsole. JXInsight also offers the ability to export metrics as offline snapshots and merge them across a cluster of managed JVM's. Additionally JXInsight provides a JVMInsight extension that inspects the complete contents (including object field state) of registered MBeanServers.
  
  JMX Related Links:
  http://www.jinspired.com/products/jxinsight/new-in-4.2.html
  http://blog.jinspired.com/?p=6
  http://blog.jinspired.com/?p=12
  http://blog.jinspired.com/?p=10

  
  William Louth
  JXInsight Product Architect
  JInspired

  Posted by: williamlouth on July 25, 2006 at 12:57 AM

• peaking in on those production servers once and a while
  You can add a fair amount of information on your boxes by installing MessAdmin. It will give you an incredible view on your applications internals!

  Posted by: applebanana8 on July 25, 2006 at 04:45 AM

• Some RMI defaults have changed in Mustang. You may want to try the same default on Tiger. Note: you need both the client and server settings.

  -Dsun.rmi.dgc.server.gcInterval=600000
  -Dsun.rmi.dgc.client.gcInterval=600000
  -Dsun.rmi.transport.tcp.connectionPool=true
  

  Posted by: alforbes on July 26, 2006 at 02:31 AM

• alforbes, the remark about RMI properties is excellent. Indeed, in Tiger, RMI's Distributed Garbage Collection would force a complete garbage collection in the server JVM every minute. This is extremely expensive! When you monitor a JVM with jconsole, the JVM becomes an RMI server, so it has this behaviour. As of Mustang, the default GC interval has become one hour.

  My measurements were too short to have run into this phenomenon, but it is worth mentioning to people who are using jconsole to monitor longer-running apps.

  I'm unfamiliar with the property sun.rmi.transport.tcp.connectionPool and don't see it in the JDK sources.
  

  Posted by: emcmanus on July 27, 2006 at 06:15 AM

• Can you tell me what is the cost if we have a thread that monitors the app performance and reports
  1)found deadlocked threads
  2) alerts if memory consumption crosses a set threshold
  NB :we are not talking of monitoring by using jconsole etc but which is passive(in my opinion)

  ours is a real time service so wanted to have these features so that the operate personnel could take corrective actions in cases like deadlock by bringing down the service etc instead of waiting for long durations etc.I'm not sure how much this is going to affect us,as this is more cosmetic and nice to have feature.so will go ahead if this is not too costly an approach.

  Posted by: minajagi on May 29, 2007 at 03:32 AM

• minajagi, I don't have a very good idea of what the overhead will be in your case. I can tell you what I would guess, but there is no substitute for measuring the overhead in your real application running in its real conditions.

  Concerning the deadlock monitoring, its overhead is probably not negligible, and JConsole does not do it as part of the periodic measurements it makes. However, it sounds as if your monitoring thread would not need to check for deadlocks very often. For example, it might be once every five minutes. If threads are deadlocked then they will stay deadlocked, so you won't miss the deadlock even if you don't check very often.

  Concerning the memory usage monitoring, you can use JMX notifications to be informed punctually as soon as the memory usage goes over a certain value. This article at gives some information about memory thresholds, and this gives the technical details of how to register for notifications. I don't know the overhead of registering but I expect it is low.

  Posted by: emcmanus on May 29, 2007 at 03:58 AM

This work is licensed under a Creative Commons License.