Skip to main content

API Design vs. API Usability

Posted by timboudreau on July 5, 2009 at 11:19 PM PDT

I took last week off to work on some Wicket web programming - seeing as my day-job is desktop programming (and also as a hedge against ending up jobless after Oracle buys Sun).

I've done a lot of talks on API design, focusing on how to not "paint yourself into a corner" in terms of backward compatibility. There is a complementary subject - API usability, that deserves equal consideration. But reconciling the two is a hard problem.

What defines a "usable" API is set more by the expectations of the users of that API - and that depends on their conception of how libraries are supposed to work - which is often at odds with how to actually make something you will be able to change in the future without breaking existing users' code. I can spell out how to create an API that you can change in the future without breaking backward compatibility with a few simple rules:

  • Make all classes final unless there are at least two clear use-cases for subclassing.
    • Where possible, keep the class final and provide an interface someone can implement and pass in for the mutable functionality
    • If you don't make the class final, make all methods final except those where you anticipate subclassing.
      • If you anticipate subclassing, all overridable methods should either be abstract or empty - people often forget to call super.foo() and you can save them that headache by simply creating abstract void onFoo(), or if you must, void onBeforeFoo() and onAfterFoo().
  • Separate API and SPI - most APIs have two sides - think JavaMail (my favorite example of adequately backward compatible design coupled with horrific usability) - there's the API that lets you talk to a mail server generically, and an SPI that lets things transparently plug in the ability to talk to an IMAP server, a POP3 server, etc. In a modular system like NetBeans or OSGi or hopefully JDK 7, the API should and SPI should be in separate packages - ideally separate JARs with separate classloaders, so even if someone wants to call directly into the IMAP implementation, they can't do so without knowing they're playing dirty tricks that are likely to make the lifetime of their software short.
    • Use the Accessor pattern to completely separate API and SPI (this requires a module system that lets you hide packages from a user of a JAR - although you can cheat somewhat by just excluding a package or two from javadoc and hoping people get the point) - i.e. you've got
      public final class com.foo.api.ImageProviderApi {
         ImageProvider (ImageProviderImpl impl) {
            this.impl = impl;
         }
         public Image getImage() {
            return impl.getImage()
         }
      }

      and
      public interface com.foo.spi.ImageProviderSpi {
         public Image getImage();
      }

      with no public constructor for ImageProviderApi - perhaps a static factory method; you use ServiceLoader or Spring or Guice or whatever to inject the implementation; the API that gets called is final.

      If you're in a modular system like the NetBeans module system, where you can hide one package, then you can use a bridge class in a hidden package to guarantee no API client can get hold of an instance of the SPI implementation (the API class has a static block that instantiates a single static instance of a subclass of the hidden accessor class that all its methods delegate to, which, in-turn, delegate to the interface - really this points to a missing language feature, IMO).

      The point of this is simple:

      • You can add methods to a final class and that is backward compatible
      • You can remove methods from an interface and that is backward compatible as long as no client of the interface can gain access to an instance of it directly. By wrapping an interface (which you can remove methods from compatibly as long as nobody can get a real instance of ImageProviderSpi), everybody gets the best of both worlds - you can remove from your interfaces (you just have to provide reasonable default handling in the wrappers), and add to your API final classes.

      The reason this is useful is that one day you may realize that it would be nice to be able to get image size without loading the image. You can do this perfectly backwardly compatibly:

      public final class com.foo.api.ImageProviderApi {
         ImageProvider (ImageProviderImpl impl) {
            this.impl = impl;
         }
         public Image getImage() {
            return impl.getImage()
         }
         public Dimension getSize() {
            if (impl instanceof ImageProviderSpi2) {
               return ((ImageProviderSpi2) impl).getDimension();
            } else {
               return new Dimension (getImage().getWidth(null), getImage().getHeight(null));
            }
         }
      }

      and
      public interface com.foo.spi.ImageProviderSpi2 extends ImageProviderSpi {
         public Image getImage();
         public Dimension getSize();
      }

      No client breakage - you just need a reasonable default implementation of the final version of the API method.
  • The amount you need to separate API and SPI and do this sort of defensive coding depends on how well-defined the problem domain is. If you are defining an interface for validating a string value, and you know that's all it will ever do, then you can safely just make an interface
    public Validator {
       /** returns a localized message of the problem if the string is bad */
       public String validate(String s);
    }

    But most problem domains people actually get asked to work on are not so well-defined - that was the curse of EJB 1.0 and other frameworks of that era - nobody knew what the web was going to really be for and so people tried to provide for every possibility - and some of those possibilities turned out to be nothing.
  • Don't try to save the world - if you are writing a validation framework for Swing components, tie it to Swing components, get the job done and move on. Don't try to create the ultimate validator of ad hoc beans and possibly reflectively their properties via property editors generated on the fly with dynamic proxies. Just get the job at hand done - lots of people will find it useful, and maybe somebody will help turn it into something that does save the world. But it's guaranteed that if you start out trying to save the world, you'll either fail or never finish.

There are a lot of other things I could talk about with regards to modular systems - for example, if your API and SPI are in different JARs, NetBeans lets you put an ad-hoc string "token" in the manifest (IMO OSGi needs this), e.g.,

OpenIDE-Module-Requires: com.foo.spi.Implementation

and there must be some other JAR in the system that says in its manifest
OpenIDE-Module-Provides: com.foo.spi.Implementation

so you guarantee some implementation of com.foo is actually loadable or the application will not start, but you don't care what implementation of the API is there - just that there is one (and matching a minimum version you can specify).

But I was talking about API usability - and the question in my mind is, are these things orthagonal or not - and can they be reconciled?

Usability, as I mentioned, rests a lot on user-expectations, and that suffers from the fact that object-oriented-programming is largely taught from the wrong perspective. People expect inheritance to be the primary mechanism for chaining logic in object oriented programming. People get taught to think in terms of structures that look like

  • Organism
    • Person
    • Cat
    • Pet

It's all very nice, hierarchical and object-oriented. But think about it:

The moment you start wondering if a Person can also be a Pet should be a big red warning flag that this is simply the wrong approach to the problem, and that any design that relies on that sort of inheritance is going to end in a swamp of ambiguity.

I used to have this four-pane slide I showed in design talks, showing me wrapping a toaster in plastic wrap, filling it with coffee, pouring water in and plugging it in. The point being that in the real world, if you try to make coffee with a toaster, you're an idiot, and in the programming world, if you succeed, you're a Rock Star. And that's not a good culture to cultivate - because if you're writing software, you're by definition doing something for someone else - would you ask your grandmother to make coffee in the toaster? If not, then that's probably not an approach you should take to software.

So, okay, those weren't such simple rules for doing backward compatible APIs, but they're not that complex either, I was just long-winded about it. My dear friend Jarda is far more eloquent and long-winded on the topic in his (worth reading) book. On to my recent experiences with Wicket:


My friend Jon created Wicket, and we've programmed together on and off since about 1982, so I am biased here. I think he has found some really good compromises between backward compatibility and usability. I have a habit of oversimplifying/misquoting him here on my blog, so I'll further that with something I think he IM'd me once upon a time that I thought hard about and think I agree with: The ideal interface has one method.

That's a really interesting idea - because even if you do break compatibility, the damage is very compartmentalized and localized and affects a minimum number of users of your API. It's not perfect, it's more like signal-theory applied to API-design - if most things don't break, then most things will be OK and that might just be acceptable.

There's a corollary to that, which is, if you need gargantuan interfaces, probably you haven't carved the problem domain up into objects the right way.

And even that has its caveats - if you're writing an interface that represents some sort of hardware device, then it should replicate that device in every detail - to do otherwise is lying, or worse, creating something broken.

See, every rule has its exceptions.

But even then, starting from the premise than any interface or abstract class should have no more than 1-2 methods users have to implement is a very useful thought-tool for taking a cold, hard look at anything you've written and seeing if it really optimally solves a problem.

The whole thing with object-oriented design is that, in the empirical sciences, we observe the world, postulate, test and build a model off of the results. In the object-oriented world, create a model of some subset of the world, give it to other people who will not empirically, but practically test it, and yell at us when it falls down.

When you are creating a library, you are playing God, literally, and there is a responsibility to make the results testable and easy to use according to the expectations of your users.

Wicket makes some interesting compromises about all of this. Someone blogged that Wicket has a "learning flat" - I tend to agree. The place where it runs smack into my backward-compatibility instincts is that if you are using Wicket, you will be programming with inner classes - all the time, all over the place. Almost any complex component has one abstract method, and you'll end up writing an anonymous inner class that implements that method, e.g.

        List auctions = new ArrayList(user.getOwnedAuctions());
        ListDataProvider ldp = new ListDataProvider(auctions);
        DataView dv = new DataView("auctions", ldp) {

            @Override
            protected void populateItem(Item item) {
                item.add (new Label("name"));
                item.add (new Label ("description"));
                item.add (new Label("voterCount"));
                item.add (new Label("state"));
                long uid = getDb().getPersistenceUtils().getUid(item.getModelObject());
                PageParameters pp = new PageParameters();
                pp.put(EditAuctionPage.AUCTION_ID, uid);
                item.add (new BookmarkablePageLink("edit-link", EditAuctionPage.class, pp));
                item.add (new BookmarkablePageLink("report-link", EditAuctionPage.class, pp));
            }
        };
        this.add (dv);

My API usability instincts tell me, "hey, this is really easy to work with!" My API design instincts tell me:
DataView should be a final class and there should be an interface (or final class w/ injected delegate interface) I should implement and pass in that provides the one abstract method.

So what's the right answer? Well, it depends. I work on NetBeans. In 1999 we got acquired by Sun (we'll see how this Oracle thing works out). NetBeans had some APIs, but Java was young, we were young, and there were plenty of ways to break compatibility we had never imagined - but suddenly we were under the same regime of absolute, perfect backward compatibility as Solaris (even though, at least at the time, NetBeans was not used to run nuclear submarines). Part of the way we learned how to do APIs backward compatibly was having to live under that regime.

But one of the key pieces of API usability is API size. The more things people see in Javadoc or code completion, the harder it is for someone who just wants to get something done to get something done. So we have a terrible choice: Should DataView (above) have a DataViewItemPopulator interface.

On the one hand, the problem with constructors and abstract classes is that you tie user code permanently to the specific type of the object.

On the other hand, with a factory method you could change the type based on what gets passed in, but something has to be passed in so you double the size of the API, and that means double the number of class names a developer has to comprehend to be able to work with it. More classes == less usability.

I think Wicket gets (or got) the balance about right (see corrolary below).

Yet there are cases where you have a moral obligation to make your API less usable. If you have an API where get() on some object might trigger long-running file or network I/O, the only responsible thing to do is to create something like

public interface ThingReceiver {
  public (Thing thing) hereYouGo();
}

and simply disallow any synchronous calls - otherwise you are lying to the users of your API. This is where the whole concept of an "enterprise java bean" was wrongheaded from the start. The JavaBean concept was a naming convention for synchronous calls to an object. One of the first errors everybody makes with their first networked app is to assume that bandwidth is free and synchronous socket calls while responding to a user are OK. The entire premise behind creating (and for this I blame the marketing guys, not the engineers) network-enabled "java beans" is wrong - as an entire industry found out the hard way.

And now a rant. I think the people currently guiding Wicket really need some experience maintaining an API (no matter how much you want to change it) backward compatibly for five or six years before doing any of the gratuitous crap I've spent the last few days dealing with in Wicket 1.4-rc5.

Re Wicket and the balance between compatibility and usability: I just spent two days updating code to work with Wicket 1.4. I am completely f**ing disgusted with the arbitrary backward compatibility breakages in Wicket 1.4. It's one thing if something is broken so badly that the only way to fix it will hurt some people. But Component.getModel() -> Component.getDefaultModel() is pure aesthetics. And stupid aesthetics at that. If you are using Wicket, you know a Component has a Model, and that model represents exactly one object. "default" adds no information value - it just breaks compatibility, hurts people needlessly, and makes them less likely to trust Wicket. You are making people do work to satisfy your sense of aesthetics. That is selfish and wrong. Really in the first place, it should have been:

public class Component<T> {
...
public abstract T get();
}

(this is Java - any class name or getter name including the word "Object" is overhead or perhaps more aptly, a semantic bug - of course an org.openide.filesystems.FileObject is an Object - there's no use in appending Object to the class name - it clarifies nothing. Same with getModelObject() - if anything it should be shortened to T get() - what's it going to return, a puppy?)

What the hell is the "non-default" model and what could that possibly mean? You still haven't released 1.4. There is time. How many Win32 applications would there be if M$ had broken their APIs between Win 95 and Win XP? Think about it. You break compatibility when you have no choice. Not when you do have a choice.

Bottom line - breaking compatibility costs people money. Lots of money. Lots of money they pay their employees to make their code work with the next rev of your code. Those employees are going to bitch about it. Hell, I'm going to bitch about it - I wasted two days of my vacation time changing "getModel()" to "getDefaultModel()" to satisfy someone's twisted need to make already too-verbose code more verbose. Whose default? What other model is there that isn't the default?

I don't care so much about the money - heck, I work for the company that loosed EJB on the world, and count the billions that sucked down a black hole. But breaking compatibility really is a breach of trust, and its a lot easier to gain trust than lose it. I'd strongly suggest putting back as much compatibility as possible before an official 1.4 release - you will only lose users otherwise.

-Tim

Related Topics >>

Comments

> Sometimes preserving API compatibility at all costs results in developers being afraid to put *anything* as a published API If it was not clear from the above, I believe netbeans has gone too far in this direction

IMHO Minor release should never break backwards compatibility, but major releases can. Code must be cleaned and refactored in order to keep it maintainable for a longer period of time. Otherwise you will drown in all the dead wood. If you make sure that many of changes are compiler detectable, and there is good migration documentation, this is simply part of the process. And quite on the opposite; I prefer to allow extending of all my classes as much as possible. I'm soo pleased Swing components are not final. This allows me -at my own risc- to modify the behavior.

> If doing so makes it easier for *me* in the future to maintain and > evolve the library, instead of cursing and abandoning altogether, > i prefer the former. I prefer it too. But the weight thousands of people depending on you not breaking compatibility is very, very heavy. I am really sorry for the problem that made substance-netbeans not remain viable, and now that I am back on the dev team, I will try to fix that. -Tim

A good read, even though i completely disagree. We all make mistakes, and instead of relying on convoluted schemes that make my life - as a developer of free / open source software that does it in his own free time - more complicated, i prefer admitting a mistake and break the API. If doing so makes it easier for *me* in the future to maintain and evolve the library, instead of cursing and abandoning altogether, i prefer the former. Sometimes preserving API compatibility at all costs results in developers being afraid to put *anything* as a published API - instead doing it in private packages accessible via reflection only. This has happened quite a few times already with Swing/Java2D/JavaFX mix. IMHO this is worse than having a public API which gets perfected - even at the cost of breaking - over time. Visit any Swing forum and you will find scores of posts from people that experienced problems migrating from older versions of Java to 6.0. Even when you preserve APIs, you break behavior. I prefer my errors being caught by compiler.

@Mikael I think we actually agree. Take a look at http://kenai.com/projects/simplevalidation/pages/Home - I got a lot of pressure from friends I asked to review it, to turn it into a framework for validation of everything and everything. I resisted that hard and kept it scoped for easy validation of Swing components. It's set up in a way that someone else can create a "save the world" framework out of it, but that's not my goal. I agree completely - scoping is key - don't try to save the world. I'll say that I don't love being lumped in with the folks who foisted EJB on the world. Yes, Sun has produced some horrendous APIs. But I sure hope that I personally have not (and I genuinely ask you to tell me if I have - self-improvement only comes from being able to take and absorb harsh criticism). @nickhomeaccount NetBeans org.openide.util.Lookup basically is IAdaptable, just older and without the assumption that what you're getting is an adapter (rather, you're requesting a known interface from a bag-o-stuff). Still, whatever interface you request from it has to have a finite number of methods, and the thing we're really after here is how to design it so that the number is at a minimum and the result is usable and understandable. Your point is entirely what this post is about. When creating an API there are a set of choices to make, and they revolve around scope. How universal a thing do you want to create? Ones' ego pushes for the save-the-world scenario - that's the thing that needs to be fought against. The core problem is that Java's design does not encourage evolvable APIs. The scoping rules it has are inadequate. -Tim

have you thought about the IAdaptable pattern?

Hello Tim, I know that you are a very smart man. But, when it comes to API I also have done quite a bit of soul searching.

I think you fail to mention the target audience of the API. I think Sun always over estimate the importance of an API. What I mean is that they make it too complicated given the target audience. Explanation:

The amount of complexity that is "just right" for an API is proportional to the number of people using it and their demography . A specific API for a specific purpose can be more pragmatic, and thus less change resilient, than a general API like the collections API.

One always need to calculate the amount of time spent in an API over time. If you create something that needs 10 minutes of extra time to understand for 10.000.000 people you have a lot of hours to account for. If the same API can be written so that the same amount of people can spent 1 minute you have gained a lot of time. Now consider the amount of time to change something, it can be a small name change or something big. If that will be 2 minutes to 10.000 people it will be several orders of magnitude smaller than making the API more change resilient to begin with. This is not counting the power of "easy to get started", which is maybe the most powerful force there is.

This is where I think Sun fails in a lot, if not all, of APIs. They always make it complex to use in the fist, simple, case, so that they can change it without being officially blamed later. IMO that is a bad way to write APIs.

When you create something, like a very flexible API, you get to be an expert very quickly. This is a problem as you think that other people also has the amount of time and interest available to spend on the problem. This is a logical fallacy though, one that is very common to programmers (and equals). You always end up being an expert at YOUR thing, but people that USE it never is (an expert).

Instead, APIs should be as simple as they can be, but with a clear and structured migration path, with a very clear target audience. This is where Swing fails, miserably, IMO.

What this mean in practice is pragmatic APIs which is dead easy to use with a clear way to proceed when the APIs change is king. Java generally has few APIs like this, there are no good way to migrate Java code.

I hope that JavaFX will have a good way to handle API changes. What I have seen so far has shows nothing of this unfortunately. I hope that this will change. I would like some language supported way to migrate API. It is all very easy, you just need to dig in,

In hope for a better tomorrow,
cheers,
Mikael Grev

> I dont agree with your getModelObject() > get() I was more suggesting IModel.T get() - it works well for java.lang.reference.Reference and its subclasses, and is clear, nice and non-verbose. I agree, Component.get() would be bewildering. getModelObject() may be the best choice for Component. I always smell a bit of stink in the air when I see any class or method named *Object* because it's usually redundant. But I agree, with Component there may have been no better alternative.

i do not think there is any one person who is completely happy with how the api changes worked out in 1.4. what has worked very well without generics in 1.3 was completely broken no matter which way we tried it in 1.4 there was no good way to genericise the default model. we actually started with component[t], but the did not work out: only a small subset of components actually use their model - if a component does not use it you are stuck with either declaring a placeholder or a warning about a missing type declaration. you would also have to declare all fields with the wildmask. all this was pure noise. Component border=new WebMarkupContainer("border"); would all of a sudden become Component[?] border=new WebMarkupContainer[Object]("border"); for every single declaration - it adds up and makes the code very noisy. once we could not genericise the component itself we had to declare the model field as IModel[?]; this presented us with another delema, the components that were generic could no longer override IModel Component.getModel() method with their own generic type - they were forced to use the wildmask. so a FormComponent[T].getModel() would have to return IModel[?] instead of IModel[T]. obviously this was a no go. We had two choices: either add a IModel[T] getTypedModel() to all the components that were genericised or rename the original IModel[?] getModel() to IModel[?] getDefaultModel(). obviously you know which way we went. the justification is that the chances of a developer calling getModel() on a non-generic component are a lot less then on a generic one, so we wanted to preserve the api for the more common case. ----------- i think the only thing that everyone could agree on was that the solution was not perfect, but it was one that made most sense and offered the easiest path forward (many users have migrated all of their codebase with a simple search and replace as far as the model changes went). tim, if you have a better idea we are always all ears. we are still looking for a way to make this better in 1.5. we do not mind breaking the api between major releases, we wish wicket to evolve and not stagnate. if we made a mistake in a previous version we are willing to fix it properly in the next release even if that means breaking the api. this has allowed wicket to stay ahead of the curve imho and i think most of our users actually like that attitude.

I dont agree with your getModelObject() > get() because looking at the api then component.get() what on earth does that return? Most of the time if wicket has get() methods like that it is a ThreadLocal like RequestCycle.get() or Application.get() So component.get() would return ....? and it is not really the object of the component it is the Models object It is just an easy helper method for developers to use If we would change i would say remove that method al togheter. so getModel() and getModelObject() it is for me logical that getModel().getObject() == getModelObject() that is not the case if it was just get()

hmm the genericst stuff where removed from te output..

Tim, The funny thing is what you say what it should have been: Component Is exactly the reason why it has been renamed!! Because soooo many people where totally disgusted by the amount of generics now had to be applied to wicket code... Thats why we "invented" it to have a default model without the need for generics. And that components that really have objects will have there generic version for example see Link that does hat the method IModel getModel() This is done all because of the horrible design of generics in java. I presented a small preso for it when we had a Wicket User Meeting (right before ApacheCon EU): http://www.slideshare.net/jcompagner/wicket-and-the-generics-battle-and-...

I work for javax.swing.dead.wood.* :-) Extending all classes looks nice on the surface. If it's strictly you're own hierarchy it can even work. But I've seen so many cases of things like class Foo extend JPanel implements PropertyEditor { private final changes = new PropertyChangeSupport(this); public void firePropertyChange(Object src, String name, Object old, Object nue) {} } where the look and feel (pure evil -running foreign code in the superclass constructor) tries to add a property change listener - and it varies by platform and look and feel - you just never know where it's going to fail next.