David Van Couvering 's Blog

October 2005 Archives

The story of the write cache and half a worm

We’ve been doing some performance testing of Derby, and we discovered something that I suspect many you out there may not be aware of. I know it caught some of us by surprise, and we’re dealing with databases all the time.

First of all, let’s talk about your data. I think most of you agree that when you store your data in a transactional database, you expect it will meet the transactional guarantees of atomicity, consistency, isolation and durability. I mean, why else go through the trouble of using a database? For example, if you commit a transaction, and your database says “OK”, then if in the next moment the database crashes, when it comes back up, the data should (a) still be there and (b) be consistent (e.g. it’s worse finding half a worm in your apple than a whole worm).

Continue Reading...

Open Source Development: Diplomacy Training

My friend has a story of how he was sitting on his deck with a beer, and he noticed a little bit of wood flaking off of a post. He got up and picked at it, and it was clear it was dry-rot. He took a screwdriver and started working it, and it went deeper and deeper. So, being a handyman, he decided to replace the post. But then he discovered that the leak had affected his foundation, and he ended up having to jack up his house and replace some of the foundation.

I thought that was a great metaphor for the task I am currently working on in Derby. All I wanted to do was internationalize the network client driver error messages. Then I noticed that there were similar error messages and SQL States already defined for the embedded driver. So I thought it would be great to share these messages between the two drivers so I wouldn't have to create new ones or cut-and-paste the existing ones (and all their translations).

Ah, the heady days of youth. Little did I know what I was walking into. The initial proposal on the derby-dev mailing list seemed simple enough. It is now a mail thread with 93 entries in it. On and on and on the discussion went. What really opened my eyes was realizing that, in the Apache open source world, you can't just have your “chief architect” make a call when there is a difference of opinion. There is no chief architect. All committers are created equal. If any committer votes down your proposed change, you can not move forward. And heaven help you if the committers start taking sides on two alternate approaches (yes, that happened). You need to have serious negotiation skills or you are at a deadlock. Consensus must be reached.

So, this has been a very enlightening experience. And actually in some ways it has helped gel the Apache Derby team. We're getting to know each other, warts and all. It's like being in a big boarding house. You all have to get along, because you're all living together. If someone starts flaming out and throwing things, nothing gets done. So you work, and you work, and, at least in this case, we finally have reached consensus (yes, there are still some nits left, but no major roadblocks). Concessions have been made, points clarified, religious doctrine has been set aside.

I think that open source veterans should be on the “A” list for countries looking for good diplomats.

Just for edification, the big sticking point in our discussion was about that old humbug of Real Product Development, compatibility. We want to allow a single VM to be using version N of the client driver and version M of the embedded driver. If both the client and embedded drivers are using the same shared component, depending on the order of your classpath, if you don't have compatibility between versions of a shared component you are going to get surprising results.

If you're curious to see how we're resolving these issues (at least, I hope so), take a look at this Derby Wiki page that summarizes the proposal.

And I haven't even broached the issue around what mechanism we should use for dependency injection in our common code. We have a service manager for the core engine, but it's pretty heavyweight for the network client. Just wait until I start that discussion! :)

This work is licensed under a Creative Commons License.