Richard Bair's Blog

Not Invented Here?

"Not Invented Here", a clever phrase often used as a pejorative to indicate that somebody (usually due to ego?) would rather reinvent the wheel and write their own implementation than leverage an existing (and by implication, better) implementation. In this entry I try to shed some light on real technical reasons why "rolling your own" may be better than relying on somebody else's implementation.

First, let me state that I am in favor of leveraging existing work whenever possible. It is often more economical to purchase tools than to build your own. It is sometimes more economical to purchase tools than use open source equivilents. It is sometimes wiser to use open source implementations than either purchasing or writing your own. Each circumstance and situation is different, there isn't one right answer for every situation.

Also, the opinions expressed below are from an API designer's perspective, not from an application developer's perspective. Some of the same principles can be applied, but it should be noted that an application developer's needs and goals are fundamentally different from those of an API designer.

I also want to go on the record and say that similar competing implementations are not always a bad thing. I agree with Romain's statement on his java.net blog:

First, creating a new and similar to another Open Source project is not reinventing the wheel. No matter how close two projects can be they'll never be exactly the same and they won't match the same requirements and goals from a user's perspective and from developers point of view. Don't think of wheels, but of tires. There are a lot of different sorts of car tires nowaday because they all suit a particular need. Every winter in France we change the tires on my dad's car to put snow tires. Why don't we complain about that and demand good tires for both steep and snowy roads as well as dry and hot freeways? Because that's not what you really want, you want a set of tires very good at one thing instead of clumsy at many purpose.

The problem with developing API's that rely on some third party project is that if the third party project decides to grow in a different direction, or modify components or APIs in a way that doess't harmonize with your own APIs, you end up having a huge mess. You end up having to fork their project (assuming that their project has a license compatible with your own). Even if they are doing fine but you have bugfixes/feature requests that go unanswered, you have to fork the project. Or when their release schedule is incompatible with your own, you end up delaying your own product or forking theirs.

This problem doesn't usually occur as an application developer, or even developing massive APIs that are internal to an organization, but happens when you are developing APIs for the general community. Also, it cannot be forgotten that portions of SwingLabs are candidates for inclusion in future versions of the JDK. As such, we need to a) have proper copyright ownership so that Sun can change the license to the standard java license and b) rewrite anything that is from a third party project anyway before it could be included in the JDK (unless we got the rights to all of their source code and liked it. Either way, it''s going to be a fork).

I can say this much, we are very interested in providing developers the best solutions possible for writing rich client apps. We're not interested in having our own personal implementation "win" (assuming there was such a thing), but rather in having the entire community win. Part of this will require many judgement calls, and where the SwingLabs team's judgement differs from another team's judgement, we'll get differences. Since SwingLabs is LGPL, there's a lot of safety in using SwingLabs -- there is no vendor lock in. And, those portions of SwingLabs that we want to move into the JDK may be part of a formal JSR as well, so everyone has a chance to have their voice heard.

Hopefully this helps present some of the trickier issues involved in using third party components/libraries, the forces and factors at work on us, and express our desire to work with the community to provide the best solution in one place, and in a way that may be accessible to all of the java community in the future.

My personal opinion is that sometimes you're better off using existing code, sometimes you're better off buying existing code, and sometimes you're better off writing your own. Which is best for a given situation is a judgement call. And hey, let's ease up a bit on the "Not Invented Here" accusations.*

[*] While this blog was inspired by zander's use of this phrase in my last blog's comments, it is in no way intended as a rebuttle or slight. Sometimes this accusation is a worthy one, and I cannot comment on whether his use of it was accurate or not. This question has been posed to me several times over the last year, so I thought now was a good time to post my opinion.

Comments

• word! nice blog'age mr bair!

  Posted by: gonzo on August 15, 2005 at 10:24 PM

• Of course your concerns mean little if there wasn't so terribly much parallel API development going on...
  With hundreds of web frameworks out there, each of them almost identical to all the others, a lot of that is pure NIH and ego building.
  Not all of it, it's conceivable (likely even) that there are scenarios not covered by any existing QUALITY product out there, but most.
  Your concerns about version hell are indeed very valid though, and a major point in my decision to not adopt products like Jakarta Commons into my own applications. These hobbyist projects (mind, I've a lot of respect for the Apache/Jakarta project in general) have APIs that change so rapidly and unpredictably that it's nigh on impossible to program against them.

  Posted by: jwenting on August 16, 2005 at 03:14 AM

• 
  I agree - good points, well made. It's so funny that I wrote something about this in a similar vein yesterday. I didn't know of Romain's recent blog either. I wonder if there's something in the air making people write about this?!

  There's a funny misconception from those who issue such complaints. The assumption is that if a project releases code under a Free license, then anyone is free to barge in and modify the source, and mould it into which ever shape they want, thus avoiding the need of two projects.

  This is quite false because licenses only apply to the end result, and not to the management and development methodolgies of the project. Many OSS projects are very open to contributions, but they often have to comply to the general vision and direction of the original project.
  

  Posted by: arooaroo on August 16, 2005 at 04:19 AM

• "First, let me state that I am in favor of leveraging existing work whenever possible. "

  The organizations I have seen that would qualify for "Not Invented Here" usually insist on doing *everything* themselves. E.G., writing their own databases, file systems, etc.

  Posted by: swatdba on August 16, 2005 at 05:34 AM

• Andy: You voice another very valid point (which I alluded to, but didn't flesh out) which is that project management is just as important as the actual source code. Look at the XFree86/Xorg fork. They have (or had) identical source bases, but very different managment philosophies. Yet another reason why two identical projects can exist, and be beneficial to the community.

  Good points.

  Posted by: rbair on August 16, 2005 at 07:07 AM

• Some time ago SwingLabs has decided to not add any dependency on outside packages to the swingx layer. And in this blog you justify that leveraging other projects can be quite an administration headache!

  But by the same token, how are you going to convince anyone to use JDNC?

  Posted by: jessewilson on August 16, 2005 at 01:06 PM

• But by the same token, how are you going to convince anyone to use JDNC?

  As I mentioned in the blog, the needs of API designers are much different than the needs of Application developers. As an application developer I've personally relied very heavily on third party code.
  I never even thought about trying to get other open source projects to rely on our code, I've always focused squarely on application developers. I may be singing a different tune for the databinding project in the future. Don't get me wrong, I'd be thrilled to see toolkits built on top of SwingLabs code, but that isn't our main goal. But again, these are judgement calls that each project has to make.

  Posted by: rbair on August 16, 2005 at 02:45 PM

• Reuse is about more than code.
  Your blog states that there is a time when you want to reuse and a time when you want to invent yourself. This is quite generic and completely true. Your blog does make me wonder if you are not defending peoples need to reinvent things too vigorously.
  In practice its almost always a net loss to start writing your own API and implementation as opposed to reuse another project's work. You should naturally be prepared to fork the work if you can't come to an agreement with the original creators. While this may be annoying, its not a problem for the long term viability of either project and surely you just got an infusion of a lot of code you suddenly don't have to write and debug yourself.
  This part I really miss from your story; when I use an already created API I instantly gain a stable API that has seen its share of debugging and all the obvious mistakes have been made and fixed. Starting from scratch you are going to make those same mistakes again. I see that in the jdnc components of the wizard, the calendar tool and the actions framework. All of which I offered from my stable open source application.
  I just smile when people think they know better then someone who has walked the path they intend to walk, and find they actually make most of the same missteps as all the people who walked the path before them. :-)
  

  Posted by: zander on August 17, 2005 at 04:21 AM

This work is licensed under a Creative Commons License.