Eamonn McManus's Blog

Eamonn McManus is the technical lead of the JMX team at Sun Microsystems. As such he heads the technical work on JSR 255 (JMX API 2.0) and JSR 262 (Web Services Connector for JMX Agents). In a previous life, he worked at the Open Software Foundation's Research Institute on the Mach microkernel and countless other things, including a TCP/IP stack written in Java. In an even previouser life, he worked on modem firmware in Z80 assembler. He is Irish, but lives and works in France and in French. His first name is pronounced Aymun (more or less) and is correctly written with an acute accent on the first letter, which however he long ago despaired of getting intact through computer systems.

JavaOne next week!

Next week is JavaOne 2008! I'll be speaking there with Jean-François Denise, about upcoming developments in JMX technology. Here are some of the other sessions you might want to attend if you're interested in that.

• BOF-5552, "Java Observability by Bytecode Instrumentation", K Balasubramainan and A Sunararajan, Tuesday 7:30pm, Esplanade 300.

  How to attach to an already-running JVM and insert instrumentation code into it. Not necessarily directly related to the JMX world but of interest nevertheless.

• TS-5199, "JMX Technology Update", E McManus and J-F Denise, Wednesday 4:10pm, Esplanade 304/306.

  That's us! I'll be talking about some of the changes to the JMX API being planned in JSR 255, and Jean-François will be talking about the Web Services Connector being designed in JSR 262.

• BOF-5698, "Practical JMX Security Options", J Gould and D Smith, Wednesday 6:30pm, Hall E 134.

  Although we've put a lot of effort into make JMX technology secure, it's always possible to get your configuration wrong and leave yourself inadvertently open. It looks as if this BOF will show some good ways to avoid that.

• LAB-1430LT, "Java Hotspot VM Trouble Shooting Tools in a Nutshell", M Li and J Shen, Thursday 12:30pm--2:30pm, Hall E 132.

  This hands-on lab is full but you might be able to sneak in. It promises to show you how to use JConsole, jps, jmap, jhat, and hprof.

• TS-6145, "Using DTrace with Java Applications: Bridging the Observability Gap", J Haslam and S Ritter, Thursday 1:30pm and Friday 2:50pm, North Mtg 121.

  DTrace is an excellent piece of technology that allows you to add probes to an application and summarize the data they provide, by writing scripts in a special language. The application itself doesn't need to be modified to add the probes. DTrace is available on Solaris 10 and on Mac OS X Leopard, at least.

• BOF-4945, "Designing Manageable Java EE Applications in a Clustered Environment", J Jensen and P Kristiansson, Thursday 7:30pm, Hall E 134.

  This looks to be a very interesting BOF for JMX weenies. The presenters are from a telecoms background but their abstract promises broad applicability.

• BOF-5223, "VisualVM: Integrated and Extensible Troubleshooting Tool for the Java Platform", L-M Alventosa and Tomas Hurka, Thursday 7:30pm, Gateway 104.

  If you haven't seen the excellent VisualVM tool, this is the occasion to discover it. A simplistic way to describe it is that it is like JConsole but with added support for CPU profiling and heap walking, and the ability to download plugins from a plugin centre. Unfortunately this BOF clashes with the previous one.

• TS-6028, "Near-Real-Time Distributed Enterprise Infrastructure for Traffic Data Collection...", J Carroll and B Smyth, Friday 2:50pm, Hall E 134.

  The JMX API is just one of a bunch of APIs mentioned in the abstract so I don't expect it to be the focus of the talk, but the subject looks pretty interesting anyway.

In addition to all that, Sun's Pod 136 in the Pavilion area will be staffed by JMX experts a lot of the time. Catch me there on Wednesday and Thursday from 11:15am to 2pm. Jean-François is likely to be around then too, if you have Web Services Connector questions. And Luis-Miguel Alventosa will be there in the same time-slot on Tuesday and Wednesday if you have questions or comments about JConsole or VisualVM.

See you there!

[Tags: javaone javaone2008 jmx]

A query language for the JMX API

The JMX API is being updated by JSR 255. That JSR is currently planned to be part of Java SE 7, and some of the API changes it defines have started to appear in JDK 7. So far, the main one is a Query Language. Here's what that is and what it's for.

The JMX API has always included the idea of queries. The idea is that you can tell the queryNames method to filter the set of objects that it returns, using an object that implements the QueryExp interface. For example, you can pick out only those MBeans that have an attribute called Enabled with the value true and an attribute called Owner with the value "Duke".

The recommended way to obtain QueryExp objects is using the static methods of the Query class. That way, your QueryExp consists only of standard classes that must be present on all JMX implementations and there are no worries about classes that are present on the client but not the server.

Up until now, the way to code the query I described above was this:

 QueryExp query =
     Query.and(Query.eq(Query.attr("Enabled"), Query.value(true)),
               Query.eq(Query.attr("Owner"), Query.value("Duke")));
    

While it's possible to decipher that and determine that it does indeed mean what I described, it isn't very easy. The idea of the query language is that you can get the same QueryExp object like this:

 QueryExp query = Query.fromString("Enabled = true and Owner = 'Duke'");
    

Much easier to understand! (Let me stress that this is just an alternative way of writing existing queries. It doesn't introduce any new types of query.)

The query language is closely based on the WHERE clause of SQL SELECT queries. Here are the other examples from the specification:

Message = 'OK'

Selects MBeans that have a Message attribute whose value is the string OK.

FreeSpacePercent < 10

Selects MBeans that have a FreeSpacePercent attribute whose value is a number less than 10.

FreeSpacePercent < 10 and WarningSent = false

Selects the same MBeans as the previous example, but they must also have a boolean attribute WarningSent whose value is false.

SpaceUsed > TotalSpace * (2.0 / 3.0)

Selects MBeans that have SpaceUsed and TotalSpace attributes where the first is more than two-thirds the second.

not (FreeSpacePercent between 10 and 90)

Selects MBeans that have a FreeSpacePercent attribute whose value is not between 10 and 90, inclusive.

FreeSpacePercent not between 10 and 90

Another way of writing the previous query.

Status in ('STOPPED', 'STARTING', 'STARTED')

Selects MBeans that have a Status attribute whose value is one of those three strings.

Message like 'OK: %'

Selects MBeans that have a Message attribute whose value is a string beginning with "OK: ". Notice that the wildcard characters are SQL's ones. In the query language, % means "any sequence of characters" and _ means "any single character". In the rest of the JMX API, these correspond to * and % respectively.

instanceof 'javax.management.NotificationBroadcaster'

Selects MBeans that are instances of javax.management.NotificationBroadcaster, as reported by MBeanServer.isInstanceOf.

like 'mydomain:*'

Selects MBeans whose ObjectNames have the domain mydomain.

If you're familiar with SQL, all of these should be familiar, except the last two, which have no SQL equivalent.

The full specification also includes a formal grammar, which I won't reproduce here. I'll just say that I got to liberate my repressed inner compiler geek when writing the parser.

Other uses for the Query Language

Apart from making it easier to write code that does queries, a standard query language is very practical for tools like JConsole or VisualVM that might want to allow the user to select a subset of MBeans using a query. A simple text field can now be used to do this.

The query language also provides one solution to a problem with the existing Query API. The methods of the Query class allow you to construct objects that implement QueryExp. But if you have such an object, there is no easy way to look inside to see what kind of query it is. As well as Query.fromString, the new API includes Query.toString to do the reverse transformation. With these two methods you can for example save a query in a text file, or send it over a text-based protocol like SOAP, and reconstruct it later.

(You might be wondering why we bothered defining Query.toString. Couldn't we just have said that the standard toString() method would do the right thing? The main reason it can't is that ObjectName is a QueryExp, but the syntax to include an ObjectName in a query is for example "like '*:type=Foo,*'", while ObjectName.toString() will be just "*:type=Foo,*".)

Custom queries

Nothing prevents you from writing your own class that implements QueryExp and giving it to queryNames. Although custom queries are powerful, they're also discouraged. Most queries that you want can be composed out of the standard set. The problem with custom implementations is that, if you want to do your query remotely, you need to arrange for the implementation class to be present on both client and server.

This is why Query.toString and Query.fromString don't specify any handling for non-standard queries. Although various possibilities could be imagined for what that might look like, people could be seduced into using custom queries without realizing the deployment headaches that that can lead to down the road.

Why SQL?

You might be wondering why the query language is based on SQL, which is a database query language. Is that really appropriate to query management objects?

Although it's far from obvious, the original JMX query API was closely based on SQL too. In fact, the places where the query language described here differs from SQL are essentially the places where the JMX query API has changed since its original version. One strong historical hint here is that the Reference Implementation has always used SQL syntax in the toString() methods of the various QueryExp classes, even going as far as to replace * and ? with their SQL equivalents % and _.

Quite apart from this, SQL is familiar to very many programmers, and is also the inspiration for the query languages used by the Java Persistence API (JPA) and the Java Message Service (JMS).

Evolution of the standard queries

The set of available standard queries has expanded slightly over time. The original set was defined in version 1.0 of the JMX API, way back in 2000.

In version 1.2 of the API, we made ObjectName implement QueryExp, which gave users a way to match ObjectName patterns themselves, and also meant that you could use queryNames to find MBeans that match a pattern AND another pattern, and various other Boolean combinations. This is the last available standalone version, and the version that was included with Java SE 5.0.

In Java SE 6 we added Query.isInstanceOf.

In Java SE 7 (assuming plausibly that that includes JSR 255, the new JMX API), in addition to the query language, we're including the ability to use dotted attribute expression like A.b.c, with the same meaning as for monitors. So for example you could find out which memory pools still have init the same as committed in their MemoryUsage using code like this:

MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName poolPattern = new ObjectName("java.lang:type=MemoryPool,*");
QueryExp q = Query.fromString("Usage.init = Usage.committed");
// or: Query.eq(Query.attr("Usage.init"), Query.attr("Usage.committed"));
Set<ObjectName> names = mbs.queryNames(poolPattern, q);

If you're sending either isInstanceOf or dotted attribute expressions over a network connection, you need to have some way of knowing that the other end supports those. Otherwise, you have to avoid using either of these.

(We could add a method, say Query.isSupported(QueryExp, MBeanServerConnection) that tells you whether the given MBean Server supports the given query, by examining its specification version. But most of the time, either you wouldn't know what to do if it returned false; or you would know what to do, and you could just do that always.)

A pattern matching problem

One area where I'm not sure the query language does the right thing is with patterns. In the existing API, if you want to query for all MBeans that have a State attribute that is a string in parentheses, you would use
Query.match(Query.attr("State"), Query.value("(*)")).
In the Query Language, you can instead write
Query.fromString("State like '(%)'").
As before, that's much easier to read, but what's the story with * versus %?

Query.match uses the well-known "shell-style" wildcards, where ? matches any single character and * matches zero or more characters. On the other hand, the LIKE operator in the SQL standard uses the characters _ and % for the same thing. Both the Java Persistence Query Language from Java EE and JMS Message Selectors have a LIKE operator that uses the SQL characters. So people familiar with these will expect the LIKE operator in the JMX query API to work the same way. On the other hand, people who are familiar with Query.match or shell wildcards or ObjectName wildcards will expect the other convention. It's particularly messy when you compare a query that matches ObjectNames, corresponding to ObjectName.apply, with one that matches strings:

QueryExp objectNameQuery = new ObjectName("mydomain:*");
System.out.println(Query.toString(objectNameQuery));
// prints: LIKE 'mydomain:*'
// this is not a standard SQL query so we don't have to respect precedent

QueryExp objectNameStringQuery =
    Query.match(Query.attr("Name.canonicalName"), Query.value("mydomain:*"));
System.out.println(Query.toString(objectNameStringQuery));
// prints: Name.canonicalName like 'mydomain:%'

Never mind the inconsistent case of LIKE (which I just noticed). Can we really live with * in some places and % in others?

This is just the beginning

You can expect other new features from 2.0 to show up in the JDK 7 snapshots in the coming months. Namespaces, Event Service, localization support, you name it!

[Tags: jmx sql jdk.]

Do I really need all those jars in my classpath?

Big applications have a tendency to accumulate enormous classpaths. Looking at such a classpath, you might be hard put to know whether any given jar is really needed. Perhaps it was needed at the time it was added, but that need has long since evaporated. How can you tell? Having jars you don't need means your application will be slower starting up, and perhaps also while running. It also means that you might be worrying unnecessarily about getting the latest version of a jar that you're not actually using.

Kyrill Alyoshin has an elegant solution using a java.lang.instrument agent. The basic idea is that you run your application with java -javaagent:loosejar.jar ... and the agent in loosejar.jar can find all the classes that the application has loaded using Instrumentation.getAllLoadedClasses(). Then for each of those classes it can find what jar it came from. Thus for each jar on the classpath it can compute the number of classes that have actually been loaded from it. When this number is zero, the jar might be unnecessary.

Of course for the results to be valid you have to exercise the application so that it does everything it can do. That might not always be easy, but for a candidate jar you can often figure out what to do to make the jar be referenced. To help you do this incrementally, the loosejar agent exports a JMX MBean that allows you to ask for a report while the application is running. So you can connect with JConsole and get a report, see what jars might be unnecessary, try to provoke class-loading from those jars, get another report, and so on.

One subtlety is that the set of jars is not just the contents of the classpath. The jar could reference other jars through a Class-Path entry in its manifest. You'd like to know if all of those other jars are really necessary too. Kyrill and I were unable to find a better way to get the transitive closure of referenced jar files than to use ClassLoader.getResources("META-INF/MANIFEST.MF") and parse the returned jar: URLs. Every jar has a META-INF/MANIFEST.MF file, so every jar known by the ClassLoader will show up in the result of this call, but ugh. There has to be a better way.

Kyrill's loosejar project is hosted on Google Code.

VisualVM - All-in-One Java Troubleshooting Tool

VisualVM is a new graphical troubleshooting tool that is being developed in a project on java.net. It gives access to the functionality that is available through existing JDK tools such as JConsole, jinfo, and jstack, and adds to that support for lightweight profiling of CPU and memory usage.

These screenshots from the project page give an idea of what the tool can do.

The tool is extensible via downloadable plugins, and the latest milestone publishes an API for developing your own plugins. Also, if you have developed plugins using the JConsole Plugin API it's possible to use them unchanged in VisualVM.

I'm not personally involved in this project but I'm certainly following it with great interest.

[Tags: visualvm.]

Speaking at JavaOne

I'll be speaking at JavaOne again this year. As in the previous two years, I'll be sharing the stage with Jean-François Denise. The main subject will be the contents of the two JMX-related JSRs that are nearing completion: JSR 255, which is defining version 2.0 of the JMX API, and JSR 262, which is defining a Web Services Connector.

Our session is TS-5199. Here's the abstract:

The JavaTM Management Extensions (JMXTM) API, part of the core Java platform since release 5.0, is being updated through two JSRs nearing completion: JSR 255 (JMX API 2.0) and JSR 262 (Web Services Connector). In this session, members of Sun’s JMX API team discuss the state of the art and explain how the new JSRs will improve scalability, ease of use, and interoperation.

Scalability covers larger applications with more managed objects and distributed management across many machines.

Ease of use covers several new annotations to make it easier to define managed objects. Annotations also make it easier to expose descriptions of those objects to management consoles, including internationalized descriptions. A new query language makes it easier to select objects that match certain criteria.

Interoperation covers management of Java technology-based applications by programs not running on the Java platform, such as scripts, or tools using the WS-Management standard. JSR 262 facilitates the integration of Java technology-based applications into heterogeneous management environments.

This session does not assume detailed knowledge of JMX technology.

In all, there are six sessions at the conference that mention "JMX" in the abstract - three Technical Sessions and three BOFs. Our Technical Session looks like the only one where JMX technology is the principal focus, but two-and-a-half of the three BOFs will be about it too. Should be interesting!

Public Review of Web Services Connector for JMX Agents

The Public Review of JSR 262, "Web Services Connector for JMX Agents", is underway, and there's a new snapshot of the Reference Implementation that corresponds to the Public Review specification.

Jean-François's blog has the full details.

[Tags: jmx jsr262 ws-management.]

May 2008

Search this blog:

Categories

Archives

May 2008
April 2008
February 2008
December 2007
October 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007
December 2006
November 2006
October 2006
September 2006
July 2006
June 2006
May 2006
March 2006
February 2006
January 2006
December 2005
November 2005
October 2005
September 2005
August 2005
July 2005
June 2005

Recent Entries

JavaOne next week!

A query language for the JMX API

Do I really need all those jars in my classpath?

This work is licensed under a Creative Commons License.