Skip to main content

Why extension methods are evil ?

Posted by forax on November 28, 2009 at 11:49 AM PST

Mark Reinhold recently blogs about the renaissance of
closure in Java (*).
In his blog, he also wrote that in order to have closures in Java,
Java will have to include another feature named
extension methods.
As you may already know, I'm a big fan of closures but
the introduction of extension methods in Java really scary me.

* Ok, they are more lambdas than closures.

Why extension methods ?

If closures are added to Java, collections API need to be retrofitted
to allow some simple things like iterate over a collection, filter a collection,
transform (map) a collection to another one using closures.


C# developers face the same problem during the development of the C# 3.0
and add a new feature called extension method.


An extension method is a static method but with a special syntax
to differentiate them from static method.
Because the syntax is not important, I use a new keyword, extension that can be used
instead of static. This allow to retrofit already existing static method to
an extension method.

  public class Collections {
    ...
    public extension <E> void forEach(Collection<? extends E> c, #void(E) closure) {
      for(E element: c) {
        closure.invoke(element);
      }
    }
  }

Now, how to use them ? First you have to include them in the current scope
of the file, C# use using for that, in Java, the equivalent is import static.
Then you can call an extension method like any non static method using dot ('.')
with the first parameter of the extension method as the receiver of the method call.

  import static java.util.Collections.*;
  ...
  List<String> list = ...
  list.forEach(#(String s) {
    System.out.println(s);
  });

Why extension methods are evil ?

As emphasis in Joe Darcy Devoxx'09 slides,
(slide 15) some of Java Language principle are:

  • Reading is more important than writing. (1)
  • The language should not hide what's happening. (2)
  • A clear semantic model greatly boost readability. (3)

Extension methods are imported to the file main scope but
nobody read import statements, IDEs hide them.
So having a way to let any users to define the semantics of functions
that operate on a very well known set of classes like the collection API is really dangerous.
This breaks principle 1.

If you add extension methods to Java, you add another way to call a method on an object.
Method calls are polymorphic in Java (VM do late binding) but because extension methods are static methods,
extension method calls are not polymorphic.
Calling extention methods with dot creates an uniform way to call two very different beasts,
not a very good idea.
This breaks principle 2 and 3.

Moreover, import static has severe limitations:

  1. You can't use more that one import static Foo.*; by file, otherwise your code is not future proof.
    Like import *, if you import static more than one classes and one of these classes has a new
    method in the future, if this method has the same name and the same number of parameter types
    than one method in another class, your code may not compile any more.

  2. local scope is more important that import static scope,
    the code below doesn't compile.

  import static java.util.Arrays.*;

  public class StaticImportAreBroken {
    void sort() {
      ...
    } 
    public static void main(String[] args) {
      sort(args); // doesn't compile
    }
  }

To finish this post with an optimistic inclination.
What we really need is an interface with code inside
and the ability to insert this extension stuff(TM)
into an already existing class that is a subtype of an interface.


Similar features already exist in lot of languages,
they are named
Mixin or
Trait
depending on the languages.
I don't advocate for any of these specifications/implementations but for
a better solution than extension methods.


Also note that
Interface injection,
one sub-project of
Da Vinci
machine project can be used to mix already existing class with extension class at runtime
thus preserving backward compatibility with already existing code bases.
I also think that this can also be implemented using the new invokedynamic bytecode
introduced in jdk7.

Cheers,

Rémi

Related Topics >>

Comments

Arguments

I think there are some good and bad arguments here and some boil down to philosophy/ideaology. One of the most compelling arguments is simply that extension methods do not fit in the philosophy of Java and that's an important thought. One of the least compelling arguments are those that say extension methods create unreadable code. I just don't buy that. Extension methods would be no more unreadable than calling a static method. As long as the extension itself is properly named it should be apparant what it does (e.g. myString = myString.encrypt("myKey"); vs. myString = encrypt(myString, "myKey);). From a management perspective I don't see how it is any more difficult to include an extension library as opposed to say, a string utils library.

I don't live in the guys of the Java source code, so I certainly can't speak to the complexity of including extensions, but as far as I understand how they work in other languages (C# namingly) it's more of a compiler trick than anything where it's passing the object to a method that resides in it's own class, hopefully that doesn't trivialize what it is doing because I clearly haven't researched it's implementation either other than seeing it in use. However, when coupled with modern IDE's like Eclipse it would make it far easier to find the methods you're looking for.

As for the backwards compatibility argument, this to me means code from jdk 1.2 will still compile in later versions. It doesn't mean that additions now will still compile on older compilers. We've got to move forward.

Too complicated

All this stuff is too complicated and sucks ! Maybe 80% of the time we need closures to implement one-method interfaces for call backs, accessing local or instance fields without declaring them final and changing their values if we wish, thats all !! I think we (java guys) waste most of our time discussing the sex of angels, and that´s why java language improvements takes so long compared to others platforms like .Net regards.

Re: Too complicated

Because Microsoft doesn't take a great care about feature combinations, by example, this doesn't compile in C#:

  var f = x => x;

Rémi

That could never possibly compile as there is nothing for ...

That could never possibly compile as there is nothing for the type inference to start with. You simply cannot do type inference like this and you need to declare the type - which in Java you can never do "var" , and always need to declare the type. So what exactly are you complaining about?

Hands off my Java, please!

Good API design is all about controlled evolution. Extension methods break that. That might please all your h4ck3s out there but for the rest of us it is a very bad idea. Secondly, it seems to me that people who are asking for "interfaces with methods" haven't had the "pleasure" of dealing with C++'s multiple inheritance. The entire point of interfaces is to avoid all the problems associated with multiple inheritance. Seriously, this discussion is making me angry. Stop proposing utterly dumb ideas just because other languages have it. There is a reason that Java is the most used programming language and those other "cool" languages have a market share of 0.001%. Please keep the focus on simplicity and readability!

Interface implementation != multiple inheritance

I've been following many of the lively debates about what should and what shouldn't be in Java 7 (or in Java Ever) and have often felt "hands off my Java" too.

In particular, I feel equally angry about the many, many fools who rant about Java "fossilising" and failing to face "The Future" which is purported to be Scala or Groovy or (Lord forbid) Ruby.

Like you, I love Java, and I love [my interpretation of] the philosophy of Java. Having looked closely at those languages and more, I only feel more and more convinced that Java is by far the greatest general purpose language for "working programmers", aka "blue collar programmers", aka "Joe Java": because it emphasises readability over writeability; because it is simple enough (bar type erasure/wildcards, sadly) to be used as a teaching language, but powerful enough to be the world's leading language for "hard core" applications of all varieties; because Java is aimed at teams, not individuals (because it's more about "say what you mean" than "compiler, mean what I say").

However, the complexities and pitfalls of multiple inheritance in C++ have absolutely nothing to do with inheriting implementations, and everything to do with inheriting data. There are absolutely no pitfalls to inheriting multiple method implementations provided that (a) those method implementations rely not on data, but on getters/setters (it doesn't matter one jot if both interfaces declare the same getters/setters), and (b) the compiler enforces no ambiguity of implemented method calls (i.e. if both interfaces implement a method, your non-interface must implement it, overriding both - usually, of course, by calling one or the other of the super-implementations).

Interfaces in Java were a great idea, because they allowed *some* of the flexibility and power of multiple inheritance, without any of the dangers of multiple inheritance. But they could have almost all of the flexibility and power of multiple inheritance, still without any of the dangers, if a proposal like this were accepted. What is good about Java interfaces has nothing to do with either (a) methods being non-implementable or (b) methods being public.

Moreover, Joe Java will have absolutely no difficulty understanding enhanced interfaces, nor experience any woe whatsoever from using them.

Please don't tar everyone who wishes to make Java even better with the brush of people who "[propose] utterly dumb ideas just because other languages have it" - though there are plenty of those about :(

Not sure I am so skeptical.

Not sure I am so skeptical. Extension methods can be overused/misused (like any other language feature) but they bring about very powerful abilities to augment and share domain specific code. It's worth noting the official definition in C#: "Extensions methods are a way of adding additional functionality to an existing class without subclassing it or having access to the source code". While I can appreciate the theoretical negative consequences, in the pragmatic world we have real problems to solve. For one example, imaging what Joda-time + extension methods could do to a gazillion code-bases all using their own buggy DateUtils.

Re: Not sure I am so skeptical

Joda-Time is not a good example, the main problem of java.util.Date/Calendar is
its interface/implementation. Extension methods will not to able to fix that.
Extension methods only allow you to add extra fonctionality if the fonctionality
can be written without modifying the existing class.

Rémi

Right, perhaps not the best

Right, perhaps not the best example. Nonetheless, how many DateUtils have you come across in your life? Extension methods makes it a lot easier to decouple context and achieve re-usability.

on extension methods

The call is not polymorphic indeed, but on the other hand if you're defining a forEach(List list) the compiler could emit a compile-time error that an extension forEach(Iterable list) is already defined in the current scope ... that, or introduce multi-dispatching (maybe through invokedynamic), which could solve a whole bag of other common gotchas.

About readability ... I don't agree. Concise is equivalent to readable in every other profession or software community, and being explicit about *everything* does tend to make the code a lot less readable because of all in-your-face explicit-although-I-couldn't-care-less boilerplate ... my current day-to-day job involves lots of Perl, and none of the projects I worked on are as bad as some of the Java-code-bases I worked on.

Java programmers tend to say that the IDE helps with writing all the boilerplate code ... but why not turn that on its head ... make the code concise and let the IDE help you read it? Let the logic of your algorithm visible, and let the IDE give you the under-the-cover implementation details?

But you're right ... having special "exceptions" to the rules are a really bad thing. The mental-model of how something works should definitely be consistent. That's why I would also prefer traits / roles through interface injection ... with clear rules for conflict resolution, such that you can't inject the same method signature from 2 roles or override an already defined method in the class. And the JVM-support would help other languages too.
Another reasonable alternative is the mechanism from Scala ... implicit conversions. When calling ...

list.foreach( .... )

... the compiler sees that "foreach" isn't defined for Iterable, then it searches for an imported class that has "foreach" with that signature, sees a "EnhancedIterable" for which an implicit conversion is defined, and the code could get compiled like this ...

EnhancedList iter = EnhancedList.convertFrom(list);
iter.forEach( ... )


The readability is OK, the IDE can help you out to see that the object you're using is an "EnhancedList", and when there are 2 conversion rules detected for that scope ... compile-time error. So no issues with dispatching the right call, no problems with static imports, and the community culture could sanction the conversion rules where information is lost (those kind of rules your daddy was telling you to avoid).

On explicit versus implicit ... explicit constructs tend to leak encapsulation. And in practice, whenever a compile-time error happens ... Java programmers tend to push that square through a round hole anyway, without thinking too much ... "oh, so you want a String and this is an Object, heh? ... here ... (String) obj".

Are extension methods really needed to make closures expressive?

Suppose we don't have extension methods, but just static methods in classes such as Collections. Then we could still write

Collections.forEach(list, #(String s) {
    System.out.println(s);
  });

or, with a static import,
forEach(list, #(String s) {
    System.out.println(s);
  });

Judging from this example, I don't think that this is a problem that calls for another language feature. And anyway, for new APIs, such as ParallelArray, this should be a non-issue, right?

Re: Are extension methods really needed...

You're right !


Don't adding extension methods is perhaps the most reasonable thing to do
but It will slower the adoption of closures unless we add a way to enhance
the javadoc format to expose methods that takes a closure as parameter.

Rémi

No brainer

I think this is a no-brainer ... interfaces should always have allowed method implementations, and it has always been an annoyance that up until now they have not.

I don't see why they should be called mixins or traits, they're just interfaces finally done right - I certainly see no need for an "extension" keyword.

While we're fixing interfaces, I wish we fix the other glaring problem - no protected methods. Daily I have to make methods public should be protected, just because they are declared in an interface.

The unfortunate original design decision to not require "public" on interface methods paints us into something of a corner here, because existing public interface methods without any modifier would of course appear to be package-private in a world where interface methods can have mixed access levels,

But I think that if people are finally to consider allowing method implementations in interfaces, it has to be addressed - or else the method implementer is forced to either write the method as one long method (rather than broken down into appropriately sized smaller method calls), or else to make all of the sub-calls which make up the method implementation public, however private (or protected) they should be.

Mark Reinhold must have not had his coffee that morning

One word: Jigsaw. If and when the api is modularized the problem is already solved, simply release a new version of the module that holds the collection classes. Afaik it was the same Mr Reinhold championing it. Same goes for protected and interfaces afaik there is going to be module public and normal public, or something along those lines.

Eh?

Um, what does Mark Reinhold have to do with my post on method implementations and access levels in interfaces? And what does Jigsaw have to do with *anything*?

RE: jigsaw

Jigsaw solve the problem of class/package/module versioning,
not the problem of binary backward compatibility.
You can't add a method to an alreday existing interface even with jigsaw.

Rémi

Use-site vs. declaration-site vs. real currying

To me, every solution using some static import to provide some kind of transparency in invoking a method on an object it does not belong to is a way to confusion. Peter Ahe has written about this shortly two years ago, suggesting declaration-site extension methods, which, I think, are what you propose here as well. In a more general post [1], I have written some thoughts on currying vs. extension methods. Currying may go a bit far, but the main point, in my view, is to drop transparency for readability.
-Stefan
[1] http://jroller.com/jadda/entry/extension_methods_vs_method_currying

Re: Use-site vs. declaration-site vs. real currying

I now remember that post from Peter Ahé, the weird thing is that
I had also blogged about it
two years ago but forget it. Btw, my position about extension methods was not too far from this post :)

About your post, currying is not used in its common sense (bind an object to a function to create another function),
anyway I prefer your syntax, i.e there is a clear difference between method call and extension method call
but it doesn't solve one problem: the ability to override an extension method with a classical method.
Iterating over a TreeSet is more efficient using a recursive function instead of an iterator,
it would be cool if it was possible to override extension method forEach for TreeSet.

Rémi

I dare to disagree

Traits (or interfaces with code, for that sake) solve a different problem than extension methods. When adding code to the collection interface traits are certainly cleaner than extensions methods. I am with you on that. However, extensions methods do have one killer feature: they allow me to add code to types that I dont own! This might not be very interesting for you guys at Sun, since you dont depend on 3rd party code. But gosh, I can tell you, there is so many 3rd party code out there (including the Java API) where some methods are missing.

PS, and also extension method can do one thing no other language feature can (besides C++ templates, of course :) and that is to define a method on a collection with a specific element type: public static void whatever(this Collection strings) { ... } very handy in a lot of domains! We did the same using #DNU in Smalltalk, and were able to refactor 20% of our domain methods to use this.

Re: I dare to disagree

You are right, extension methods allow anyone to virtually add any methods to any existing classes,
this is really powerful but the drawback is that these kind of methods allow you to create
kind of domain specific languages that nobody will be able to read that and figure out what the hell is doing.
That why I think extension methods are evil according to Java philosophy and principles.

Extension methods are not a killer feature but a killing feature :)

PS: I'm not a Sun guy and I don't think that you can write any Java code without using a 3rd party lib,
This is why I like Java, there is an open source community behind that provide great frameworks and libs.

Rémi

Considering that extension

Considering that extension methods have a receiver, they have a reasonably narrow context in every call site so I don't think the problem is severe. You can already abuse the existing static-import feature for the same effect... and static-imported methods can't have a receiver, which makes them even worse. You can abuse annotations to build hermetic DSLs, even changing the semantics of code with elaborate bytecode rewriting (not recommended, but then, people don't always follow recommendations). You can even make this kind of abuse without changing the language, just put loads of important behavior in XML files and use the SpringFramework, he he. ;-) Other upcoming Java7 features, like JSR-292 support syntax and even exotic identifiers and ARM, will have some potential to be abused to produce "write-only" code. Now don't throw the baby out with the baby water. Of curse, if we find a baby that makes the water a little less dirty without major compromises, that's always good.

OK for me (at least as a compromise)

I agree that mixins are a better solution than extension methods, but mixins are a much bigger language change (new typesystem feature) so i don't think it's realistic to expect this in JDK7. It's easy to get carried by the idea that JDK7 was delayed so we have time to add more features, I'm personally crossing my fingers for some extras (like tail calls), but OTOH I don't want that JDK7 slips even more.

Having said that... I don't agree with your argument wrt reading, because IDEs can fix this problem trivially. When I edit Java code my IDE will format uses of static fields/methods with italics ("semantic highlighting"), and it's trivial to add a different formatting, e.g. underlined, for uses of extension methods. Modern language syntax design should not be restricted by the needs of Notepad/vi users. (In that case Java would already be hopeless, or any modern language for that matter as we need so many APIs so we can't live without code completion and other smart helpers.)

And I don't agree with the argument of source compatibility either, because this is an extremely small problem (not different than other potential source-breaking changes that we accepted in the past, e.g. enum keyword). But the main counter-argument is that static import already incurs the risk you describe even without extension methods, unless we freeze the static part of every API's public interface. People who worry too much about this can simply avoid wildcards in static imports; just static-import each needed method or field individually. As you said, IDEs will do these imports automatically and they will also hide the imports so we're no bothered by a page-long import list.

Not Ok for me :)

First, I don't want mixin, I want interface with code. You can imagine something really simple
that will not change the type system:

  • method of interface can have code if they are marked with keyword extension.
  • if a class implements two interfaces with extension methods with the same signature,
    you have to provide a new code

  • I.super.m() means calling the code of extension methods m of interface I
interface Collection&lt;E&gt; {
  public boolean isEmpty();
  public int size();
  ...
  public extension void forEach(#void(E) closure) { ... }
}

This simple proposal requires small changes in the compiler
and I think less changes than if C# method extensions are implemented.
It also requires to changes the way interface tables are populated in the VM, that's all.

Rémi

Looks much better

This is a design I didn't consider before, pretty good. It doesn't completely replace extension methods because I can use the latter to add my own method to Collection, which is often useful - for such "open" APIs like collections, there's always hundreds of useful algorithms that won't be included in the core, not to mention application-specific utilities.

Well, I'll be happy enough with interfaces with code, but I'd like a more generalized implementation of that. The current restriction of no implementation code in interfaces should just be lifted. The extension keyword is not necessary; if you find a method with implementation, then it's an extension method - and the whole concept is only useful to javac internals, language users shouldn't be bothered with this except in the special case of ambiguity due to multiple inheritance (and btw, I suggest that such ambiguity is resolved automatically in the common case of repeated inheritance).

Static initialization blocks should also be allowed, in fact static code is already allowed for initialization of fields to arbitrary expressions, but I miss static blocks for very complex initializations, e.g. for a complex collection that must be inited from data parsed from a config file. The current design, that allows code in field initialization but not in static blocks, is absurd IMHO.

This post explains pretty

This post explains pretty well while I'm so worried about JDK7 after the latest Devoxx news. You know that I've always been against the introduction of closures, not because I don't think they are useful, but because I think that Java has reached its stability peak. Now, the idea of a "reduced closure / lambda" that emerged from the Devoxx announcement I'm less worried because they might be a reasonable compromise (I'm worried for another aspect, that is the messy announcement and the lack of clarity about what will actually make its way to JDK7 - I fear everybody's seeing his own wishful thing here). Still, there's the domino effect that you mentioned, that is the introduction of a given feature, that has been designed to do the less harm, will induce the introduction of other features that are a mess. Indeed extension points are a mess - @opinali, you know that I'm a IDE-guy, but I don't think that to solve the complexity of a language issue we can resort to the IDE. I mean, one thing is the IDE resolving some boilerplate code issues (such as generating getters and setters), but the IDE can't resolve a conceptual mess that is having two kinds of method calls, apparently the same, but one polymorphic and the other not, as Remi explained.

That's why I'm worried. Also because now the discussion about these things will go in a hurry. Remi solution about interfaces with code is probably the less-harming one, but still I fear we're really in shortage of time for analyzing all the side effects that might arise.