The Attach API lets you discover and attach to the Java VMs running on your local machine. JMX Cascading lets you federate several JMX agents together. Can we combine the two?

This is a question we've had fairly often, and I was prompted to write about it after Nilesh Bansal posted a question to the JMX forum in the Sun Developer Network to which one possible answer is to combine Cascading with the Attach API.

The solution I propose here assumes that you are running on JDK 6. If you are stuck on JDK 5, I have a few suggestions below, but there is inevitably a loss of functionality.

The basic idea

The Attach API is what JConsole uses to list the local Java processes in its Connect dialog. It is a nonstandard but supported API, which means it should be present on any JDK-derived VM that is at least version 6.

Once you select a Java process to connect to, JConsole again uses the Attach API to find the address of the JMX agent within that process, so it can connect. And, thanks to some deep magic, JConsole can use the Attach API to create a JMX agent within the process if it doesn't already have one.

So if we grab the code out of JConsole that does all this, we can combine it with Cascading. The picture below is my standard Cascading picture. Here, we're going to create a Master Agent that you can use to see the MBeans in all of your local Java processes. The Subagents in the picture are these local processes, and all of their MBeans are imported into the Master Agent. So if you connect JConsole to the Master Agent, then you can also see what's going on in each of the Subagents. In other words, you can use one JConsole connection to see all of your Java processes on that machine.

If you set up the Master Agent to be remotely connectable, then you can also see all of your processes from a JConsole running on another machine. The easiest way to do this is with -Dcom.sun.management.jmxremote.port and related properties. You need to pay attention to security in this case, of course. You probably don't want just anyone to be able to see and modify the MBeans of your processes.

Here's what we'd like JConsole to look like when it connects to the Master Agent:

In the picture, you can see cascaded MBeans from processes 16872, 18213, and 29375. Through the "Cascader" MBean, which I'll define below, you can see what command each process id corresponds to. 29375 is the command com.sun.enterprise.server.PELaunch, which is the Sun App Server. I launched that because it has a non-trivial number of MBeans, as you can divine from the MBean domains on the left.

The processes you see are your own processes. In other words they are the ones that you are able to manipulate if you are logged in to the machine. On Unix machines, this means they are processes that are running with the same user id as the Master Agent. On Windows machines, it means processes that you have the necessary privileges to open and interact with.

The details

I'll use the Cascading implementation from Open DMK, the Open Source version of the Java Dynamic Management Kit (Java DMK) product. (When I wrote my earlier blog entry about cascading, Open DMK hadn't yet been released.)

So to try this out, you'll need to download the Open DMK binaries. You'll need to compile and run with OpenDMK-bin/lib/jdmkrt.jar from the Binary Zip. The Binary Zip also includes API documentation if you want to see the details of the API I'm using.

You'll also need to compile and run with tools.jar from the JDK to get the Attach API. It's in the lib directory of your JDK 6 installation.

The program creates a "Cascader" MBean, which has one operation and one attribute. The refresh operation gets the current list of Java processes and sets up cascading to import from each one. I don't attempt to detect when new processes are created, though it would be straightforward to call refresh periodically in a loop. When a process exits, the Master Agent's connection to it will break, and Cascading will automatically remove the MBeans that were imported from that process.

The CascadeResults attribute shows the result of attaching to each Java process - either an exception message, or a success message including the process's command-line parameters. In the latter case you can use this to figure out what the process is. That's how I knew that 29375 was the App Server, above.

(A bug in JConsole makes it hard to see the command-line when it is long. If you let the mouse hover over the value, you can see a longer string.)

Since the full code is fairly involved, here's a sketch of how I discover the Java processes and set up cascading for them. It omits exception handling and the like.

List<VirtualMachineDescriptor> vmds = VirtualMachine.list();
for (VirtualMachineDescriptor vmd : vmds) {

    String id = vmd.id();
    // This is the id that we will prefix to MBean names.  In practice
    // it is a numeric process id, as we have seen.

    VirtualMachine vm = VirtualMachine.attach(vmd);

    Properties agentProps = vm.getAgentProperties();

    String connectorAddressProperty =
    	"com.sun.management.jmxremote.localConnectorAddress";
    if (!agentProps.containsKey(connectorAddressProperty)) {
    	// The above property will be present if the process was launched
    	// with -Dcom.sun.management.jmxremote or if this program or
    	// JConsole previously attached to it.  Otherwise we need to start
    	// the management agent within the process:

    	String javaHome = vm.getSystemProperties().getProperty("java.home");
    	String agent = javaHome + "/lib/management-agent.jar";

    	vm.loadAgent(agent, "com.sun.management.jmxremote");
    	// This has the same effect as running with -Dcom.sun.management.jmxremote

    	agentProps = vm.getAgentProperties();
    }

    String connectorAddress = agentProps.getProperty(connectorAddressProperty);
    JMXServiceURL url = new JMXServiceURL(connectorAddress);

    CascadingService cascade = ...;
    cascade.mount(url, connectOptions, objectNamePattern, id);

    vm.detach();
}

The complete code is below. You'll probably want to customize it, for example so it only cascades from certain of your processes rather than all of them.

Cascade loops

One detail I omitted from the sketch is that we have to be careful that the Master Agent doesn't try to import its own MBeans. Say the Master Agent is process 12345. It will show up in the list of Java processes from the Attach API, along with all the others. If we handled it the same way as the others, then the MBean java.lang:type=Runtime, for example, would be imported as 12345/java.lang:type=Runtime. Cascading detects when new MBeans are created, and imports them. So it would detect the new 12345/java.lang:type=Runtime and import it as 12345/12345/java.lang:type=Runtime. And so on.

The way I prevent this is by checking the MBeanServerId property of the MBean Server Delegate in each subagent. If it is the same as the MBeanServerId of the Master Agent, then we know that the supposed subagent is actually the Master Agent.

A more efficient technique would be to compare the VM id against the RuntimeMXBean's Name attribute in the Master Agent. But that would require making assumptions that go beyond what is strictly specified.

Stuck on JDK 5?

If you're currently on JDK 5, the Attach API is one of the many goodies that may encourage you to migrate to JDK 6. But what if you can't?

In that case, I think your best bet is to look at the JConsole code from the JDK 5 sources, and try to do the same thing it does. The relevant code is in the method getManagedVirtualMachines() within the class sun.tools.jconsole.ConnectDialog. This solution is fragile, because nothing prevents an update of JDK 5 from changing how this works and breaking your code. But if you're prepared to live with that, this is a possible approach.

Since JDK 5 doesn't have Attach On Demand, you will only be able to cascade processes that were explicitly run with the -Dcom.sun.management.jmxremote option. I don't see any way around that.

The code

package net.mcmanus.eamonn.cascadelocalvms;

import com.sun.jdmk.remote.cascading.CascadingService;
import com.sun.tools.attach.AgentInitializationException;
import com.sun.tools.attach.AgentLoadException;
import com.sun.tools.attach.VirtualMachine;
import com.sun.tools.attach.VirtualMachineDescriptor;
import java.io.File;
import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.SortedMap;
import java.util.TreeMap;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.management.JMX;
import javax.management.MBeanServer;
import javax.management.MBeanServerConnection;
import javax.management.MBeanServerDelegate;
import javax.management.MBeanServerDelegateMBean;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

public class Cascade {
    private static final String localConnectorAddressProperty =
            "com.sun.management.jmxremote.localConnectorAddress";
    private static final Logger logger = Logger.getLogger(Cascade.class.getName());

    public static void main(String[] args) throws Exception {
        MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();

        CascadingService cascade = new CascadingService(mbs);

        CascaderImpl cascader = new CascaderImpl(cascade);
        cascader.refresh();
        mbs.registerMBean(cascader, new ObjectName(":type=Cascader"));

        System.out.println("Ready");

        Thread.sleep(Long.MAX_VALUE);
        // If this thread exits then the app will exit, so don't let it.
    }

    public static interface CascaderMXBean {
        public void refresh();
        public SortedMap<String, String> getCascadeResults();
        // key in map is VM id (typically a pid)
        // value is result of trying to cascade from that VM
    }

    public static class CascaderImpl implements CascaderMXBean {
        private final CascadingService cascade;
        private SortedMap<String, String> cascadeResults;

        CascaderImpl(CascadingService cascade) {
            this.cascade = cascade;
        }

        public void refresh() {
            uncascadeJVMs(cascade);
            cascadeResults = cascadeJVMs(cascade);
        }

        public SortedMap<String, String> getCascadeResults() {
            return cascadeResults;
        }
    }

    private static SortedMap<String, String> cascadeJVMs(CascadingService cascade) {
        logger.fine("Cascading JVMs...");

        SortedMap<String, String> results = new TreeMap<String, String>();

        List<VirtualMachineDescriptor> vmds = VirtualMachine.list();
        for (VirtualMachineDescriptor vmd : vmds) {
            logger.fine("Attaching to " + vmd);
            String id = vmd.id();
            VirtualMachine vm;
            try {
                vm = VirtualMachine.attach(vmd);
            } catch (Exception e) {
                logger.log(Level.FINE, "...exception attaching", e);
                results.put(id, "Exception attaching: " + e);
                continue;
            }

            // From this point on we must detach, so we need a 'finally' block
            try {
                Properties agentProps = vm.getAgentProperties();
                if (!agentProps.containsKey(localConnectorAddressProperty)) {
                    logger.fine("...loading management agent into JVM");
                    try {
                        loadManagementAgent(vm);
                    } catch (Exception e) {
                        logger.log(Level.FINE, "...exception loading agent", e);
                        results.put(id, "Exception loading agent: " + e);
                        continue;
                    }
                    agentProps = vm.getAgentProperties();
                }

                String addr = agentProps.getProperty(localConnectorAddressProperty);
                if (addr == null) {
                    logger.fine("...still don't have connector address??");
                    results.put(id, "No connector address even after loading agent");
                    continue;
                }

                JMXServiceURL url = new JMXServiceURL(addr);
                logger.finer("...connector address is " + url);

                if (isThisJVM(url, cascade.getTargetMBeanServer())) {
                    logger.fine("...is local JVM");
                    results.put(id, "Local JVM");
                    continue;
                }

                Map<String, ?> connectOptions = null;
                ObjectName pattern = new ObjectName("*:*");
                String mountTo = id;

                try {
                    String mountId =
                            cascade.mount(url, connectOptions, pattern, mountTo);
                    logger.log(Level.FINE, "...mounted with id " + mountId);
                    results.put(id, "Success: " + vmd.displayName());
                } catch (Exception e) {
                    logger.log(Level.FINE, "...exception mounting JVM", e);
                    results.put(id, "Exception mounting JVM: " + e);
                }

            } catch (Exception e) {
                logger.log(Level.FINE, "...unexpected exception", e);
                results.put(id, "Unexpected exception: " + e);
            } finally {
                try {
                    vm.detach();
                } catch (IOException e) {
                    logger.log(Level.INFO, "Could not detach vm " + id, e);
                }
            }
        }

        return results;
    }

    private static void uncascadeJVMs(CascadingService cascade) {
        String[] ids = cascade.getMountPointIDs();
        for (String id : ids) {
            try {
                boolean unmounted = cascade.unmount(id);
                if (!unmounted)
                    logger.info("Was not mounted: " + id);
            } catch (IOException e) {
                logger.log(Level.FINE, "Exception unmounting " + id, e);
            }
        }
    }

    private static void loadManagementAgent(VirtualMachine vm)
    throws AgentLoadException, AgentInitializationException, IOException {
        String javaHome = vm.getSystemProperties().getProperty("java.home");
        String agent = javaHome + File.separator +
                "lib" + File.separator + "management-agent.jar";
        File f = new File(agent);
        if (!f.exists())
            throw new IOException("Management agent not found: " + agent);
        agent = f.getCanonicalPath();
        logger.fine("...load management agent from " + agent);
        vm.loadAgent(agent, "com.sun.management.jmxremote");
    }

    private static boolean isThisJVM(JMXServiceURL url, MBeanServer mbs) {
        try {
            JMXConnector jmxc = JMXConnectorFactory.connect(url);
            MBeanServerConnection mbsc = jmxc.getMBeanServerConnection();
            MBeanServerDelegateMBean myDelegate =
                JMX.newMBeanProxy(mbs, MBeanServerDelegate.DELEGATE_NAME,
                    MBeanServerDelegateMBean.class);
            MBeanServerDelegateMBean remoteDelegate =
                JMX.newMBeanProxy(mbsc, MBeanServerDelegate.DELEGATE_NAME,
                    MBeanServerDelegateMBean.class);
            boolean same = myDelegate.getMBeanServerId().equals(
                    remoteDelegate.getMBeanServerId());
            jmxc.close();
            return same;
        } catch (Exception e) {
            logger.log(Level.FINE, "Cannot determine if same JVM", e);
            return true;  // if it really IS the same JVM, then we might
                          // get an infinite cascading loop
        }
    }
}
// END OF CODE

[Tags: jmx, jdmk, opendmk.]