Tim Boudreau's Blog

Tim Boudreau had his first startup when he was 13, and has been hooked since, with brief departures to play rock and roll, write and play music and do graphics and photography. He is the coauthor of NetBeans, the Definitive Guide from O'Reilly and Associates. Tim was part of the team that open sourced NetBeans, Sun Microsystems' Java(tm) development environment, and currently work as a developer on that project. Originally from Massachusetts, he lives in his adopted home city, Prague, Czech Republic.

Nobody Thinks They're An Enterprise

So, today's the day I publish this rather benign rant, and either ruin my career, or more likely, wonder, "what was the big deal about this anyway?". I promise I'll get back to talking about software architecture soon :-)

The question was: "Can I use the Enterprise Pack for NetBeans if I'm not an enterprise?" The answer is, of course you can! The fact that the question had to be asked is a symptom of a disease endemic to the technology industry. It can be summed up in one word: Enterprise.

My first close-up encounter with the enterprise cargo-cult was in 1999, when NetBeans was first acquired by Sun. Some folks came out to Prague, and kept talking with fear and awe about enterprise customers (best spoken in dramatic basso profondo, ala James Earl Jones or similar). The impression it made was that an enterprise customer is most like an irate two-year-old with teething pains, a baseball bat and the muscles to use it. I'd never heard of these terrifying enterprise customers before - I'd generally just been dealing with people in our beta program who worked for big companies and gave us great feedback. But apparently the greatest horror that could befall you was to talk to your customers - because then one day Scott McNealy might be out playing golf with the president of Smrdistan, and he'd say "I need you to fly that programmer, Jan Pičovak, out to fix my technology problem!" and then you'd lose one of your best programmers for a month. This was such a horror that the best way to avoid it was to make sure programmers never actually talked to the people who use what they make. It would have all been funny if it weren't so very, very weird.

I thought to myself, how did these people stay in business? They think our customers aren't people.

Enterprise is a market segment. The thing about market segments is that you (as someone marketing something) segment the market. That is, you're the one with the knife carving up the world into segments and naming those segments. The problem happens when you start thinking the rest of the world knows your names for them, or that you are talking to a segment, not to actual people. And the height of silliness is to think that the names you made up for market segments will actually mean something to the people in those segments.

What I don't get is how this point is nearly completely lost on the technology sector. Google sometime for enterprise software. Contemplate the amount of money spent on all the web pages and sponsored ads you see. Have you ever been looking for a piece of software or technology and that was what you searched for? Ever wake up one morning and say to yourself "Gee, I think I'll get some enterprise software today! Dunno what it's for, but I'm an enterprise, so it's the software for me!"? I bet you looked for something related to an actual problem you wanted to solve.

But what is a poor marketing drone to call these potential customers (bear in mind I've been a marketing drone, but hopefully not too drone-like and never for something I didn't believe in)?

Enterprise works. It has heft. It's one word. It sums things up nicely. But it's a segment. It's your name for a bunch of people. Toyota doesn't make a car called a Toyota Soccer Mom. What an unbelievable opportunity they're missing...not.

One of the best essays or statements I've ever read is Zed Shaw's Indirection is not Abstraction. That's dead on. The (basso profundo) enterprise is a layer of indirection - it's a thing that makes it harder to talk to people. The confusion is mixing that up with an enterprise being an abstraction for (and thus an okay substitute for) people.

Now, in any other industry, you'd go out and figure out how to reach those people and if you need to adjust what you make or how you make it or how and where you communicate to reach them better, you'd do that. In technology, you go up on the rooftop and shout EnterpriseFooFaw and wonder why all the cars in the street didn't just screech to a halt. And if they don't screech to a halt, you go out of business. Or go find a job at the next bit of fresh technology meat that comes along and run it into the ground too.

Soccer Mom is a useful market segment. It's also a near-insult. There's a reason we will never see the Toyota SoccerMom or the Ford SoccerMom driving down the street.

Java Studio Creator was a good tool. But it was specifically targeted at something called corporate developers (code for people who program in Visual Basic), and so was the marketing campaign. Way to lose your audience. I once asked a room of about 30 tech writers to raise their hands if they were a corporate writer. Big surprise, nobody did. Could somebody find commonalities among those people such that they'd probably really have a use for, or imagine they had a use for, specific products? Sure! Would insulting them to their faces help sales? Probably not (I got in big trouble in that preso for a slide with corporate developer surrounded by a big red circle with a slash through it - there were people who actually thought telling people they were corporate developers was a Good Idea™. While we're at it, why not substitute drone for developer? - but that brings us to a whole other topic, which is how corporate developer plays well to "analysts" [i.e. the people who determine your stock price] even if it makes no sense for your customers - which is its own industry-wide problem - talking to corporate analysts might be one step closer to talking to your customers, but there are a few thousand steps that come next).

I've seen quite a few successes and failures in technology marketing over the years. The successes tend to be targetted at people and specific problems; somewhere deep in the Matrix-like enterprise fantasy pod farm is a human being who will decide to use or not use what you make. We're not talking to the Borg Collective here. The spectacular failures tended to be cases where (aside from genuinely lousy technology - there are plenty of ways to blow it) they were promoted to the vendors' fantasy of their customers, rather than to the actual people who actually are trying to solve a problem the technology would solve, and who were capable of making a decision.

We can't control stupid marketing campaigns, but we can constrain them. So here's Tim's Rule #1: Any technology whose name contains the words business or enterprise is crap until proven not. And might be crap even when proven not.

If enough people take that to heart, maybe we'll see a few less really stupid marketing campaigns.

People make technology; people use technology. It's really people all the way down. And that's the "message" of this blog: Nobody thinks they're an enterprise.

Writing Portlets in Wicket using the OpenPortal Portlet Server

I have patched OpenPortal to implement the apache portal bridge properly (add a class that assigns the ThreadLocal inside com.sun.portal.portletcontainer.appengine.PortalAppEngineServlet.service() and unassign it on exit; then implement the bridge calling that (I'm sure this is not how the OpenPortal folks would prefer such a thing to be implemented - the Portlet API appears to have been written to take great pains not to assume servlets are part of the equation, and that's probably a Good Thing™, but here it's needed.

The result is almost working - still have to figure out why OpenPortal sometimes sends an ActionResponse where it looks like it should be a RenderResponse; that and completely grokking how org.apache.portals.bridges.common.PortletResourceURLFactory is really supposed to rewrite URLs (looking at how Apache Jetspeed does it didn't help as much as I'd hoped) should do it.

So, sorry for the false alarm, but given that my instructions involved patching a bunch of stuff (now it involves patching OpenPortal, Wicket and Apache Portal Bridges - it's been a long day of reading source code for multiple portal servers - I expect nobody went off and immediately tried to duplicate this.

For anybody concerned about it, the 80/20 rule still stands :-)

I'm helping my friend Jon on a project involving Portlets. There is a nice integration of OpenPortal (the Sun Java System We Made This Name So Long You Can't Say It Portal Server by any other name) with NetBeans, for one-click deployment. But OpenPortal is missing the Apache Portlet Bridges stuff necessary to use the wicket portlet. So I wrote an implementation last night. It's not beautiful, and is currently tied to Wicket, but it will get you going.

It's a fun project also because Jon and I grew up a mile from each other and learned programming together. Interesting that we're doing today (sitting in front of a computer coding together) exactly what we were doing 28 years ago (except that we were about 12 and it was Z-80 assembly language).

Anyway, he's trying to get some stuff going with Wicket and Portlets. I've got to say I'm not in love with the Portlet spec to say the least - for some light reading check out the WSRP spec (although if a portal server implements it, what it does could be cool). I do like the idea of writing Portlets as simple Wicket Applications - that is sweet.

Anyway, here's what you need to do to make it work. It goes through Wicket to get the filter, which is probably not really what you want, but it does make it usable (if someone knows a way to get this stuff from inside the bowels of OpenPortal and implement it right, let me know).

First you need to get the sources to the Apache Portals Bridges project. Don't get scared - we're just patching one file to fix a bug in the common subproject, and it's simple. Open org.apache.portals.bridges.util.PortletWindowUtils.getPortletWindowId. If you have NetBeans Maven Support, you can just open the project, look up the type and rebuild it.

getPortletWindowId() does something odd: It calls session.getAttribute(PORTLET_WINDOW_ID) and casts the result as a String. Then it puts a Double under that key. Needless to say, if this method is called twice in the same session, it will throw a ClassCastException, which is what we're fixing. Change two lines and add one so it looks like this:

        Object portletWindowId = session.getAttribute(PORTLET_WINDOW_ID);
        if ( portletWindowId == null )
        {
            synchronized (session)
            {
                Double value = new Double(Math.random());
                portletWindowId = value;
                session.setAttribute(PORTLET_WINDOW_ID, value);
                Enumeration names = session.getAttributeNames(PortletSession.APPLICATION_SCOPE);
                while (names.hasMoreElements())
                {
                    String name = (String)names.nextElement();
                    if (PortletSessionUtil.decodeAttributeName(name).equals(PORTLET_WINDOW_ID) && 
                        value.equals(session.getAttribute(name,PortletSession.APPLICATION_SCOPE)) )
                    {
                        portletWindowId = name.substring("javax.portlet.p.".length(),name.indexOf('?'));
                        session.setAttribute(PORTLET_WINDOW_ID, portletWindowId);
                        break;
                    }                    
                }
            }
        }
        return portletWindowId.toString();

Anyway, it's all in the attached project - get that and build it and put it on the classpath of your web application. It contains an interface your Wicket application needs to implement (provides public access to WicketFilter->ServletContext) and a subclass of WicketPortlet [or you can implement IPortletBridge from the attached project on your existing Application if you're already subclassing it - this is the ugly stuff]). Similarly, your WicketApplication subclass needs to implement IWicketApplicationPortletBridge (just makes getWicketFilter() public).

Finally, modify your portlet.xml to point to my subclass of WicketPortlet (I said this was ugly, didn't I?) or your portlet if it now implements my IPortletBridge interface)

<portlet-class>com.sun.portal.bridge.BridgePortlet</portlet-class>

Now you need to add two parameters to your portlet.xml to tell it where the implementations of the two Apache Portals Bridges implementation are:

<init-param>
    <name>ServletContextProvider</name>
    <value>com.sun.portal.bridge.ServletContextProviderImpl</value>
</init-param>
<init-param>
    <name>PortletResourceURLFactory</name>
    <value>com.sun.portal.bridge.PortletResourceURLFactoryImpl</value>
</init-param>

If you've already got your Portlet working on, say, JetSpeed, you won't need to modify the web.xml file for your web application. If you're just getting going, here's mine for completeness:

<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">
    <session-config>
        <session-timeout>
            30
        </session-timeout>
    </session-config>
    <welcome-file-list>
        <welcome-file>index.jsp</welcome-file>
    </welcome-file-list>
    
    <filter>
        <filter-name>WicketPortlet</filter-name>
        <filter-class>org.apache.wicket.protocol.http.WicketFilter</filter-class>
        <init-param>
            <param-name>applicationClassName</param-name>
            <param-value>com.jons.project.JonsWicketApplication</param-value>
        </init-param>
        <init-param>
          <param-name>detectPortletContext</param-name>
          <param-value>true</param-value>
        </init-param>        
        
    </filter>
    
    <filter-mapping>
        <filter-name>WicketPortlet</filter-name>
        <url-pattern>/wicket/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>INCLUDE</dispatcher>
    </filter-mapping>    
</web-app>

The implementation of the bridges interfaces can be downloaded here - it's just an Ant-based NetBeans project. Add it to your web app's classpath and implement the two interfaces described above. A zip of the sources is also available (you need portals-bridges-common, Wicket 1.3.x and the servlet API on the classpath to build it).

Objects not Strings - transparent serialization over key/value pairs with proxies

Probably someone is already doing this - there are certainly enough serialization libraries out there. But here's the thought - I was thinking of URL parameters when doing this:

Say you have some parameters and you are expecting something like "id=128,name=foo,showDetails=true" - in some format like a Preferences object, a Wicket PageParameters or anything else that parses or aggregates key/value pairs. The InvocationHandler for the proxy transparently reads and writes the values based on the name of the method called (i.e. getFoo() means return the value for "foo" for whatever the storage is.

It's much nicer to work with objects than this sort of thing directly. So what if you could do something like this:

public interface Params {
   public int getId();
   public boolean isShowDetails();
   public String getName();
}

Params params = new MySerializationStrategy().read (params);

Poof - a bunch of ugly code becomes a nice friendly object. It has a few caveats (for example, an integer returning method with no key in the data needs to return something like -1).

It's probably not so useful if there is one place the object with the key/value pairs is used. But if the result needs to be processed by multiple things, passing around an object is much nicer, and saves the code being littered with magical string keys. There is the price of a little reflection overhead, but there are cases where it would be useful.

I wrote a little proof of concept over Preferences objects which works nicely; you could do this over anything that boils down to key/value pairs. I wonder if the Wicket guys would be interested in a patch to PageParameters to do this.

Semi-checked exceptions?

I'm out in Seattle visiting my friend Jon - he has bad RSI right now, so I'm helping him out on a project. He told me an interesting idea he was kicking around when he was at Sun in 1997 or so, about how checked and unchecked exceptions should have been done. It needs thinking through, but I think it has some merit.

There are problems with some exceptions - both historical ones - things that should have been checked but aren't, and things that aren't but should have been. For example:

• NumberFormatException — This is an interesting one. It's not checked. In some applications it probably shouldn't be - the application can go merrily on. In other cases you actually want to handle it
• MalformedURLException — Every time a MalformedURLException is thrown, God kills a kitten. Seriously, this is an example of hyper-aggressive validation that probably made sense in HotJava, but outside writing a browser, this just clutters people's code horribly.
• DataObjectNotFoundException — in nine years of writing NetBeans modules, I have never seen this thrown, but my code is littered with tests for it.

Jon's idea is simple and elegant: Allow individual exceptions, by type, to be flagged as checked or unchecked depending on a flag set on the package (if unspecified, you get the default). I.e. the compiler would understand this, and not compile something if an exception could be thrown, but if it had been flagged as unchecked, allow it to compile. Sort of like turning on and off assertions by package, but at compile time.

Anything like this would need a lot more thinking through (for example, if you call into another package with a contradictory setting to yours, what happens?).

But it's an interesting thing to think about (even if it will probably never happen for Java). What do you think?

Simple Dependency Injection with ServiceLoader in JDK 6

There are a lot of dependency injection/IOC frameworks out there. What you may not know is that there is a very simple yet useful one built into the JDK. And it's type-safe.

JDK 6 introduces ServiceLoader. ServiceLoader loads things based on flat files in the directory META-INF/services from the Java classpath. The mechanism of using META-INF/services has been around since JDK 1.3, under the name the Java Extension Mechanism. ServiceLoader makes it actually easy to use.

For those familiar with NetBeans Lookup API, ServiceLoader is like a severely crippled version of that (specifically the default lookup). But what ServiceLoader does, it does adequately. See below for details of the differences.

Using ServiceLoader is simple - like the capability pattern, it is based around querying for a type with a Class object and getting back zero or more instances (which may be subclasses) of that type.

Here is a toy example of using ServiceLoader for injection:

package serviceloaderdemo;
import java.util.ServiceLoader;
public abstract class HelloProvider {

    public static HelloProvider getDefault() {
        ServiceLoader<HelloProvider> ldr = ServiceLoader.load(HelloProvider.class);
        for (HelloProvider provider : ldr) {
            //We are only expecting one
            return provider;
        }
        throw new Error ("No HelloProvider registered");
    }

    public abstract String getMessage();

    public static void main(String[] ignored) {
        HelloProvider provider = HelloProvider.getDefault();
        System.out.println(provider.getMessage());
    }
}

package serviceloaderdemo;
public class HelloImpl extends HelloProvider {
    @Override
    public String getMessage() {
        return "Hello World";
    }
}
      

serviceloaderdemo.HelloImpl
      

Let's look at a more real example. A couple of blogs ago I showed a cleaned up design for something like SwingWorker. It has an interface called TaskStatus with a bunch of setters which a background task can use to update status information, presumably in a UI of some kind. What would be very nice is if someone could provide a UI for showing progress of background tasks just by putting a JAR on the classpath, and fetching a component and adding it to their window. So it's the perfect case for dependency injection — somebody should be able to write such a UI implementation, and then just put it on the classpath, and an application can pick it up and use it without changing a line of code, and it can be reused across many applications with almost no code added to those applications.

We are going to need a factory for TaskStatus objects, because there might be more than one background process runnning at a time, and the UI should show all running background tasks. So we'll create two classes, using the mirror-class pattern discussed in my previous blog - we will separate the API and SPI. The UI will get a ProgressMonitorFactory and get the UI component to add to its status bar by calling getCapability(Component.class). All ProgressMonitorFactory will do is sanity check arguments and thread state, and then delegate to the implementation which is looked up using ServiceLoader.

But there's an added wrinkle here: There are at least two flavors of status UI we will want - one which just displays in the statusbar, and one which blocks the UI with a modal dialog.

That's okay: ServiceLoader can provide us with more than one implementation - we just create multiple implementations, and add more lines to that file in META-INF/services.

So, to indicate different kinds of UI, we'll use annotations, and look up an instance using strings (we actually don't want type-safety here - someone might want to provide still another kind of progress monitor UI - but we will define two defaults):

@Retention (value=RUNTIME)
public @interface MonitorKind {
    public static final String DEFAULT = "default";
    public static final String BLOCKING = "blocking";
    String kind();
}
  

public abstract class ProgressImplFactory {
    public abstract ProgressImpl create (Task task);

    public <T> T getCapability (Class<T> type) {
        return null;
    }
}
  

public final class ProgressMonitorFactory {
    private static final Set<String> notFoundKinds = new HashSet<String>();
    private final ProgressImplFactory delegate;

    private ProgressMonitorFactory(ProgressImplFactory p) {
        this.delegate = p;
    }

    public ProgressMonitor create (Task task) {
        return new ProgressMonitor (delegate.create(task));
    }

    public static ProgressMonitorFactory getInstance(String kind) {
        if (kind == null) {
            throw new NullPointerException ("Kind null");
        }
        for (ProgressImplFactory p : ServiceLoader.load(ProgressImplFactory.class)) {
            //Would be nicer if ServiceLoader let you get the type
            //without creating an instance, ala NetBeans Lookup
            Class type = p.getClass();
            MonitorKind mk = (MonitorKind) type.getAnnotation(MonitorKind.class);
            if (mk != null && kind.equals(mk.kind())) {
                return new ProgressMonitorFactory(p);
            }
        }
        if (!notFoundKinds.contains(kind)) {
            notFoundKinds.add (kind);
            Logger.getLogger(ProgressMonitor.class.getName()).log (Level.FINE,
                    "Requested ProgressMonitor kind '" + kind + "', but it is" +
                    " not registered on the classpath.",
                    new IllegalArgumentException());
        }
        return null;
    }

    //implementation omitted...
}
      

Now we just need a JAR that registers an actual implementation of ProgressImplFactory on the classpath, which provides a UI component. All an application needs to do to use it and have a nice UI for background tasks is

ProgressMonitorFactory factory = ProgressMonitorFactory.getDefault();
Component statusComp = factory.getCapability (Component.class);
if (statusComp != null) {
    statusBar.add (statusComp);
}
      

The main point here is that the JDK has a useful built-in dependency injection mechanism, and that you can create APIs and implementations of them that are a joy to use and easy to update by using it. While it's not as rich as some of the dependency injection frameworks out there, it has the benefit of being type-safe, and is more than enough for many purposes.

Why You Might Want To Use NetBeans Lookup Instead of ServiceLoader

The differences between Lookup and ServiceLoader are as follows:

• You can't instantiate a ServiceLoader yourself; you can use Lookup for inter-object communication, not just loading services from the classpath, and it is very useful for that
• You can listen for changes in a Lookup - really, subscribe to changes the presence/absence/identity of a particular type in that Lookup. That means, if the classpath changes dynamically, you will actually be able to unload and reload objects. This makes it possible to hot-reload JARs without restarting the application.
• You can find out if a Lookup contains an instance of a type without actually instantiating an object of that type. With ServiceLoader you can only query by actually instantiating objects.
• You can use Lookups.forPath(String) to register instances in other subdirectories of META-INF. This makes some interesting things possible, such as using path information as metadata. For example, if you wanted to register different implementations of the same type for different MIME types, you could simply call

              Lookup lkp = Lookups.forPath ("text/plain");
              SomeObject thingy = lkp.lookup (SomeObject.class);
            

• You can compose multiple lookups together with ProxyLookup, and even change the set of lookups being proxied on the fly and get appropriate events.

The Capability Pattern - Future-Proof Your APIs

Here is a simple pattern which you can use to make your APIs extensible, even by third parties, without sacrificing your ability to keep backward compatibility.

It is very frequent to create a library which has two “sides” — an API side and an SPI side. The API is what applications call to use the library. The SPI (Service Provider Interface) is how functionality — for example, access to different kinds of resources, is provided.

One example of this is JavaMail: To read/write email messages, you call JavaMail's API. Under the hood, when you ask for a mail store for, say, an IMAP mail server, the JavaMail library looks up all the providers registered (injected) on the classpath, and tries to find one that supports that protocol. The protocol handler is written to JavaMail's SPI. If it finds one, then you can fetch messages from IMAP servers using it. But your client code only ever calls the JavaMail API - it doesn't need to know anything about the IMAP service provider under the hood.

There is one very big problem with the way this is usually done: API classes really ought to be final in almost all cases. SPI classes ought to be abstract classes unless the problem domain is extremely well-defined, in which case interfaces make sense (you can use either, but in a not-well-defined problem domain you may end up, over time, creating things with awful names like LayoutManager2).

I won't go into great detail about why this is true here (my friend Jarda does in his new book and we discuss it somewhat in our book Rich Client Programming). In abbreviated form, the reasons are:

1. You can provably backward compatibly add methods to a final class. And if the class is final, that fact has communication-value — it communicates to the user of that class that it's not something they might need to implement, where an interface would be more confusing.
2. You can backward compatibly remove methods from an SPI interface or abstract class, if your library is the only thing that will ever call the SPI directly is your library. Older implementations will still have the method, it just will never be called (in a modular environment such as the NetBeans module system, OSGi or presumably JSR-277, you would enforce this by putting the API and SPI in separate JAR files, so a client can't even see the SPI classes).

Given those constraints, what happens if you mix API and SPI in the same class (which is what JavaMail and most Java standards do)? Well, you can't add methods compatibly because that could break subclasses. And you can't remove them compatibly, because clients could be calling them. You're stuck. You can't compatibly add or remove anything from the existing classes.

As I've written elsewhere, it is the height of insanity that an application server vendor is supposed to implement interfaces and classes that its clients directly call — for exactly this reason. It would be much cleaner, and allow Java APIs to evolve much faster, if API and SPI were completely separated.

But part of the appeal to vendors, for better or worse, to implement these specifications, is that they can extend them in custom ways that will tie developers who use those extensions to their particular implementation. This behavior not entirely about being evil and locking people in. There is a genuine case for innovation on top of a standard - that's how standards evolve, and some people will need functionality that the standard doesn't yet support.

Enter the capability pattern. The capability pattern is very, very simple. It looks like this:

public <T> getCapability (Class<T> type);
      

Now, what can you do with it? Here's an example. In my previous blog I introduced an alternative design for how you could do something like SwingWorker. It contains a class called TaskStatus, which abstracts the task status data from the task-performing object itself. It is a simple interface with setters that allow a background thread to inform another object (presumably a UI) about the progress of a task.

In light of what we just discussed, TaskStatus really ought to be a final class. So let's rewrite it a little, to look like this. We will use a mirror-class for the SPI.

public final class TaskStatus {
    private final StatusImpl impl;
    TaskStatus (StatusImpl impl) {
        this.impl = impl;
    }

    public void setTitle (String title) {
        impl.setTitle (title);
    }

    public void setProgress (String msg, long progress, long min, long max) {
        //We could do argument sanity checks here and make life
        //simpler for anyone implementing StatusImpl
        impl.setProgress (msg, progress, min, max);
    }

    public void setProgress (String msg) {
        //...you get the idea
        //...
}

public abstract class StatusImpl {
    public abstract void setTitle (String title);
    public abstract void setProgress (String msg, long progress, long min, long max);
    public abstract void setProgress (String msg); //indeterminate mode
    public abstract void done();
    public abstract void failed (Exception e);
}
      

So we have an API that handles basic status display. But people are going to invent new aspects to status display. We can't save the world and solve everybody's task-status problems before they even think of them - and we shouldn't try. We don't want to set things up so that it's up to us to implement everything the world will ever want. Luckily, it doesn't have to be that way.

Since we've designed our API so that it can be compatibly added to, we let the rest of the world come up with things they need for displaying task status, and the ones that a lot of people need can be added to our API in the future. The capability pattern lets us do that. We add two methods to our API and SPI classes:

public abstract class StatusImpl {
    //...
    public <T> T getCapability (Class<T> type);
}
public final class TaskStatus {
    //...
    public <T> T getCapability (Class<T> type) {
        return impl.getCapability (type);
    }
}
    

Let's put that to practical use. Someone might want to display how much time remains before the task is done. Our API doesn't handle that. Through the capability pattern, we can add that. We (or anyone implementing StatusImpl) can create the following interface:

public interface StatusTime {
    public void setTimeRemaining (long milliseconds);
}
      

public T runInBackground (TaskStatus status) {
   StatusTime time = status.getCapability (StatusTime.class);
   for (...) {
      //do some slow work...
      if (time != null) {
          long remaining = //estimate the time remaining
          time.setTimeRemaining (remaining);
      }
   }
}
      

Even better, our Task API is, right now, not tied specifically to Swing or AWT - it could be used for anything that needs to follow the pattern of computing something on a background thread and then doing work on another one. Why not keep it un-tied to UI toolkits? All we have to do is make the code that actually handles the threading pluggable (I'll talk about how you do this simply using the Java classpath for dependency injection in my next blog). Then the result could be used with SWT or Thinlet as well, or even in a server-side application. Instead of a SwingWorker, we have an AnythingWorker!

But we know we need a UI - and we know we are targetting Swing right now. How can we really keep this code completely un-tied from UI code and still have it be useful?

The capability pattern comes to our rescue again - very very simply. An actual application using this UI simply fetches the default factory for StatusImpls (you need such a thing if you want to run multiple simultaneous background tasks and show status for each — my next blog will explain how this can be injected just by putting a JAR on the classpath) and does something like:

Component statusUi = theFactory.getCapability (Component.class);
if (statusUi != null) {
    statusBar.add (statusUi);
}
      

If you are familiar with NetBeans Lookup API, the capability pattern is really a simplification of that (minus collection-based results and listening for changes).

The point here is that the capability pattern lets you have an API that is composed completely of nice, future-proofed, evolvable, final classes, but the API is extensible even though it is final. The result is that the API can evolve faster, with fewer worries about breaking anybody's existing code. Which reduces the cycle time to improve existing libraries, and all our software evolves and improves faster, which is good for everyone.

It also helps one to avoid trying to “save the world” — by allowing for extensibility, it is possible to create an API that is useful without needing to handle every possible thing anyone might ever want to do in that problem domain. Trying to save the world is what leads to scope-creep and never-finished projects. In this tutorial I discuss the don't try to save the world principle in a practical example.

Does the mirror-class design seem a bit masochistic? I think it does point up a weakness in the scoping rules of the Java language. It would definitely be nicer to be able to, on the method level, make some methods visible to some kinds of clients, and other methods visible to other kinds of clients. But regardless of this, it's even more masochistic to end up “painted into a corner,”^[1] and unable to fix bugs or add features without potentially breaking somebody's code. That's how you end up with ten-year-old unfixed bugs.

^[1]painted into a corner &mdash An English idiom meaning to leave yourself with no options — you were painting the floor of a room in a pattern such that you end up standing in an unpainted corner of the room, and you can't leave the corner until the paint dries.

Where's the state?

Where's the state? This is a small but useful question when deciding how a problem domain gets carved up into objects: What things have state? What things have values that can change? When and how can they change? Can the changes be observed? Who needs to observe changes in state?

These questions make a good start for figuring out how to carve up a problem domain into objects, if you observe the principle keep all related state in one place. I would go so far as to say that the majority of the appeal of Model-View-Controller (MVC) architecture is that it encourages you to keep state in one place.

One big reason why statefulness matters is threading. In practice, it is common to see designs where threading was an afterthought. This is to be expected. Concurrency is a not natural way for human beings to think. So commonly a library is designed with no thought about threading; then a problem shows up which multi-threading can solve. Threading ends up being introduced into a codebase that was never designed for it. Retrofitting a threading model on something designed without one is very painful and sometimes impossible.

Some threading models can be simple. For example, any time you are writing a Swing application, and you have a method that can do long running I/O, the first line of that method should be

assert !EventQueue.isDispatchThread();

But keeping I/O off the event thread is the trivial case. Managing mutations to a model inside an application is a more complicated, less obvious problem.

Any time you find yourself writing a setter (or more generally, mutator) method, it is worth asking yourself Who does this piece of state really belong to? Am I putting state in the right place? Blindly following the beans pattern (add a getter/setter to whatever type you happen to be editing) can result in state being distributed all over objects in an application. The result will be harder to maintain, and if it has any lifespan, less and less maintainable over time. Like entropy, or the no broken windows theory, the more state is distributed all over the place, the more that will masquerade as design and make it okay for a programmer to further diffuse state throughout objects in the application. Human beings can only hold so many things in their minds at a time. The more state is diffused, the more things someone must pay attention to simultaneously to solve a problem in that code.

Another question above was Who needs to observe changes in state? There are some corollaries: Where should those changes be published? and Does the thing observing the change need a reference to the thing that changed?

Let's return to the concept of having a model for a minute. A model gets all related state in one place quite well. It does nothing for threading issues — these still have to be handled somehow within the model, or appropriate use of the model in a multi-threaded environment must be documented. There is a thing you can do at this juncture that can make life much easier — and it is all about how the code that uses the model is designed. If you're creating an API, there is probably more to it than just a data model. There is also how the model is accessed.

Essentially what you can do is this: If you design the rest of your API so that the only way to get a reference to the model is to be passed one (i.e. “don't call us, we'll call you”), then you know the context (particularly, on what thread, while holding which locks) in which your code is going to be called.

I'll pick a class at random as a case in point: SwingWorker. While useful, it does represent poor design in a couple of ways:

• It mixes the concerns of task state (progress, title) and performing a task and the outcome of that task in a single class.
• It uses the beans pattern in something that is obviously not a JavaBean.

I mentioned mixing concerns: Avoiding mixing concerns is not just an academic issue of idealized architectural purity. APIs are things human beings use. If you avoid mixing concerns in a public class, you can name things more clearly, and the result will be easier to understand and need less documentation (which most people won't read unless they have to anyway).

In a perfect world, a thing representing a task should probably be stateless (subclasses can have state if that makes sense for them - but no setters or getters or mutable state in the interface/superclass). If the code that does the work is passed a reference to a model object it can set task state on, that untangles the task-state concern from it.

It also removes a bit of psychological weirdness: Someone who is writing a SwingWorker subclass being is required to think to themselves “Now I have to tell myself what my state is.” I suspect many developers anthropomorphize classes they are writing — they think of the class they are editing, or the program counter, in the first person, as I, myself. I don't have empirical evidence for this, but I know I do it, and I have heard many other developers who were discussing code do it - if there's someone out there who emphatically does not do anything of this sort, I'd be interested to know about your thinking style when coding - please speak up.

Detangling the other concern is as simple as returning a java.util.concurrent.Future from the method that enqueues a task. The result would look something like this:

public interface Task<T> {
    public T runInBackground (TaskStatus status);
    public void runInForeground (TaskStatus status, T backgroundResult);
}

public interface TaskStatus {
    public void setTitle (String title);
    public void setProgress (String msg, long progress, long min, long max);
    public void setProgress (String msg); //indeterminate mode
    public void done();
    public void failed (Exception e);
}
        

and then perhaps you would have a concrete TaskRunner class that runs tasks

public final class TaskRunner {
    public <T> java.util.concurrent.Future<T> launch (Task<T> task, TaskStatus statusUI) { ... }
}
        

or, perhaps preferably, Task is an abstract class with a static launch(Task) method (or instance method - this is a matter of taste) and the TaskRunner class is an implementation detail. (Further refinements would be to look up an injected implementation of TaskRunner, and/or a factory for TaskStatuses on the classpath, etc., not to mention making TaskStatus a final class that delegates, but that's getting out of scope...)

What's the difference between this and SwingWorker? First, we've detangled all of the concerns and made them explicit. While sometimes wadding up a bunch of related concerns may seem like it's doing someone a favor, consider that you really need to read the documentation to figure out what a SwingWorker does. While the above design is simpler, you could probably figure out how to use the above API without reading a line of documentation and get it right.

More importantly, by passing in a TaskStatus instance, we've encouraged healthy client code — no code can get a reference to the passed TaskStatus until the Task is actually being run, and no code can get a reference to the passed TaskStatus unless the client actually wants it to. Contrast this with SwingWorker, which (!!!) allows foreign code to call firePropertyChange() and which can set its status to "done" or "started" inside its constructor! We haven't guaranteed correct behavior (a client could pass a reference to a TaskStatus to some object that will try to do something evil with it on another thread, although implementations could defend against this). But we have strongly encouraged proper usage by designing in such a way that the TaskStatus is only made available within the scope where it is actually used. In addition, since we're the one passing in the reference to the TaskStatus instance, we can much more safely assume what thread will call the TaskStatus instance, and what locks are held when it does. While deadlocks will surely still be possible, we're doing as much as possible to discourage them on our side.

One of our questions above was Does the thing observing the change need a reference to the thing that changed? In this case, absolutely not. In fact, to make the Task object available could only do harm. There is no reason for Task's run*() methods to be exposed to code that are interested in the status of a Task, not the Task itself. It causes harm not only because status-reporting code could potentially invoke run*() methods, but because, to the programmer who is interested in the status of a Task, the run* methods are simply noise. Most developers learn APIs through code-completion in their IDE; mixing concerns just makes it harder to figure out what is and is not relevant.

You'll notice that we've completely removed use of the listener pattern here. The listener pattern is seriously overrated, but is often used by default because developers have seen it in the Java core libraries — we use what we know. The listener pattern has a number of problems. If you've ever had to debug a Swing application where you have a severe bug that only occurs after you switch Windows from Classic to XP appearance or vice versa, then you know this pain — I know I do. What is happening is:

• Before the look and feel change, the look and feel's listeners on components get called before the application's listeners
• After the change they get called after the application's listeners

Especially when you have a situation where it is very unlikely ever to have more than one listener, the listener pattern is probably not a good choice. Listeners monitor changes in state. It is not good design to assume that multiple things will want to monitor the status of a Task without any evidence to back up that assumption If needed, it is easy enough to provide a ProxyTaskStatus (and note here the "listeners" are provided as a constructor argument - any ordering issues are those of the caller)

public final class ProxyTaskStatus {
    private final TaskStatus[] statii;
    ProxyTaskStatus (TaskStatus... statii) {
        this.statii = statii;
    }
    public void setTitle (String title) {
        for (TaskStatus status : statii) { 
            status.setTitle (title); 
        }
    }
    //...
}
        

if that's really a need; or leave that up to someone implementing TaskStatus (it can always be added to the API later if there are sufficient use-cases).

What we have, instead of the listener pattern, is a situation where we're passing in the model object (in this case, the thing that holds state about status), as opposed to having the state belong to the task itself and having other code listen for changes in the state. Most likely that model object writes directly to a UI showing task progress. I believe I've read of this pattern being called don't call us, we'll call you somewhere (if anyone can point me to where I ran across that usage, let me know so I can give credit where it's due).

To get back to threading for a minute: There are three ways I typically see threading + statefulness dealt with in code:

• Close your eyes and pray — i.e. do nothing with regards to threading and hope there is never a problem (or better, document that the code is intended to be single-threaded and the caller needs to enforce correct threading behavior because you don't)
• Synchronize everything — make deadlocks (you hope) somebody else's problem — you pay a price with synchronization — even if not so much with performance, with maintainability
• Enforce an actual threading model

This last item brings me to my main point here: The combination of keep all related state in one place plus don't call us, we'll call you creates the possibility of encouraging and even actually enforcing a threading model over a complex data model, without (generally) having to document lots of rules about threading and pray that people obey them. A weakly enforced example of this from the JDK is Document.render(), where you pass a Runnable that is guaranteed to block any other threads writes until the Runnable has exited (it is weakly enforced because you can read the document's content outside of this method — and there is much code out there that does so).

Lastly, for anybody about to scream that I have misrepresented SwingWorker here (it has lots of complexity about batching up interim results and so forth, and the above is a simplification), I'll mention that

1. I don't mean to ding the authors of it - they were being consistent with the library they were working in. Nonetheless, one could do all of those things with the above general design, and have a much simpler, easier-to-use result.
2. I could have chosen many APIs to pick on here. I work for Sun. It seems more polite to call your own baby ugly than call someone else's baby ugly :-)
3. Patterns are easier to abstract than anti-patterns. Anti-patterns tend only to become really visible after code containing them has been in production for a while. There's lots of things we know now about what not to do that simply weren't known a decade ago.

API Blogs

One thing which I think about often is the design of code, software libraries and APIs. I've been working on deriving some principles from the things I do intuitively based on experience. Whether those are useful to anyone else is an open question. Peer review is the best tool for figuring out if these really make sense or not, so I'd appreciate feedback on my next few blogs - hopefully one day they can make up some articles or a book or similar. Whether the results are of value is not for me to judge. If, at worst, what comes out of this is a healthy debate and identifying some things that are not good ideas, that has value too. Why is a subject like API-usability interesting? Well, the history of software is a history of libraries. Libraries have APIs. An operating system is a set of libraries. The Apache web server is a set of libraries. The Java platform is a set of libraries. The set of things people simply take for granted when they write code are things provided by libraries. If you're writing code, you're usually creating an API whether you know it or not. Better API design skills on average means more software that people can use and rely on faster. There are plenty of prescriptive books on coding - Effective Java is one of the best, and I recently reread it. I do hope some prescriptions can come out of this writing; what I miss in prescriptive books is the sort of “Here's how to think about such and such a problem in order to arrive at a good solution” advice that helps one to know when one has encountered the illness the prescription cures. A good little computer scientist is very tempted to avoid anything that isn't more-or-less mathematically provable, or at least demonstrable from statistical evidence. But objects are a convenience for human beings! There is a psychology of how you carve a problem up into objects, how you name things, and so forth. Some ways will be easier to use for the majority of people than others. Yet, at the same time, design is a matter of balancing sometimes fuzzy principles of code ease-of-use with the requirement that the result ask only a sane amount of work from the computer, and sometimes you still have to let the computer win. Take, for example, the argument against mixing concerns in a single class. There are excellent arguments from the perspective of architectural purism, and from the perspective of code reuse, for why mixing concerns is a bad idea. What I seldom see articulated is the simple notion that if you mix concerns, people will have a harder time understanding how to use what you create. You can't write a proof for that - and there will be cases where mixing concerns actually is the optimal solution. Those are the dark waters I would like to venture into.   There is much fascinating research to be done in the area of api-usability. To be done properly, that should involve not only programmers and usability experts, but cognitive psychologists. It would be wonderful to have such research, and would also take years to do, and I suspect cognitive psychologists have their hands full with problems more interesting to them than how to design an API. In the absence of such research, perhaps some thoughts from one humble programmer will stir up some wisdom.   The problem with writing this sort of thing is that often you cannot write about single patterns or principles in isolation - the point is how a number of patterns interact. This inevitably makes the subject complex. For example, this first of these, which I will post shortly, touches on Statefulness and Model Driven Architecture The Don't Call Us, We'll Call You principle The listener pattern being more harmful than good, and Threading models. Discussing each of these in isolation might be interesting, but it wouldn't provide the “here's a useful way to think about this class of problems” aspect — and that's the thing I really feel is missing from most design prescriptions - prescriptions are the “what”. The “when”, “how” and “why” are at least as interesting as the prescription itself. Hopefully your feedback, flames and/or howls of protest will help sort the relevant from the irrelevant. Don't be sparing - if I don't post at least one thing that is approaches dead-wrong, I'm not thinking hard enough.

Stern warning in the University of the Philippines Computer Science Building

Manila; Moderation done right; Breadcrumbs for NetBeans

My friend Geertjan and I just did NetBeans Day Manila, in the Philippines, followed by two days of plugin-writing training at the University of the Philippines. The students were wonderful and bright and motivated and it's always a joy to teach to people who are really interested in what you have to say.

There's a conversation going on on the mailing list for bloggers on this site - there's been a big issue with people posting comments that consist of 500 porn URLs and such. Back when we were little-bitty NetBeans-the-tiny-Czech-startup we had this solved on our mailing lists - 100% spam-free - a lot better than I can say for how things were after we joined Sun. The system was incredibly simple. My friend, and later, roommate for two years Karel Zatloukal, came up with this brilliant and blindingly obvious system - his becoming a manager was a huge loss to hackers everywhere. It's dead-simple:

• Any incoming message is delayed for 30 minutes
• There is a "hot-list" who get messages un-delayed
• Any "hot-list" member can bounce a message they deem to be spam

This seems like such an obvious solution. There are legitimate objections to this approach for mailing lists, although it worked well for us - the immediacy of conversation is reduced, because any incoming message will reach its recipients only a half-hour after it was sent. But for blog comments? It seems pretty ideal.

Anyway, I wanted to share that approach so that it would not be lost to history. It worked really, really well, and I highly recommend it. In our case it was a copy of majordomo hacked within an inch of its life, but the model is simple and easily repeated on any similar system. I feel like most moderation systems for mailing lists and blog posts are still stuck in the stone-age, and I've experienced vastly better almost ten years ago.

Lastly, at the booth in Manila last week, someone commented to me about a new "breadcrumb" feature in IDEA or Eclipse that they wished NetBeans had. I thought, gee, that would take about 20 minutes to implement, and set about doing it. Well, 20 minutes to get the rough draft. Another few hours to really make it useful. Geertjan already blogged about it (do a text search for "breadcrumb"). It's now available for NetBeans 6.0 and 6.1 on the plugin portal at this URL and should be available in the 6.0 and 6.1 built-in plugin managers soon.

September 2008

Search this blog:

Categories

Archives

August 2008
June 2008
April 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
February 2007
January 2007
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
March 2006
February 2006
January 2006
November 2005
October 2005
September 2005
August 2005
July 2005
June 2005
April 2005
March 2005
February 2005
January 2005
December 2004
September 2004

Recent Entries

Nobody Thinks They're An Enterprise

Writing Portlets in Wicket using the OpenPortal Portlet Server

Objects not Strings - transparent serialization over key/value pairs with proxies

This work is licensed under a Creative Commons License.