David Walend's Blog

Sharpening the Axe or Shaving the Yaks?

And I started thinking through what sorts of tools might fit the problem well. Since we don't know what sorts of data structures we'll need, but do need to manage a lot of data early, XML:DB and xml binding technology looks like a good fit. JaxB 2.0 isn't ready, XStream comes close to doing what I want, XPath 2.0 isn't available yet, etc. I'm considering asking for time at work to pour into open source efforts. And I can switch the project to JDK 5, and swap out the source code control system to Subversion, and pretty print the whole works while I'm moving things around.

I took a break to discuss these ideas with a respected colleague. He said, "Be careful. You're talking about shaving a lot of yaks."

Yak shaving is what James Gosling is worried about in this blog:

"But for me, the big problem with "axe sharpening" is that it's recursive, in a Xeno's paradox kinda way ... to sharpen the axe, you need a sharpening stone. ... But to get there, you need to build a dog sled...."

But to use a dog sled I need snow, so I need to go to town to get a snow cone machine. I grab my trusty yak to help you haul the machine from town. But it'll be summer before I get to town, and I don't want the yak to get to hot, so I shave the yak. In mid February, I proudly lead my shining, bald, shivering yak into my quarterly progress review...

Yak shaving is downloading an old JDK and rearranging an open source project's code to run inside of Eclipse's debugger to fix a problem of automatically generating ant build.xml files to build a searchable version of your source code to help find cut-and-paste sections of code with a thready bug. When you need to upgrade your linux to get a new glibc, you should smell fresh Essence du Yurt aftershave.

It's yak shaving when the people you work for have no hope of fathoming what you're doing. Sharpening an axe is fine; someone asked you to chop down trees, so explaining that you need a good sharp axe is easy. When they catch you shaving the yak, they get mad and you get embarrassed. Yak shaving is something to avoid, and something I think I'm wandering towards.

Now for some technical content: Has anyone out there used a nice, well-heeled XML-database subsystem? How do things like Xindice and Xstream play with each other? Any way to make an XML database aware of java inheritance without waiting for XPath 2.0? Or is there a better general way to take on this sort of problem without "X" anywhere?

Meanwhile, my yak is looking a little stubbly. I better lather him up.

Comments

• The balding yak stuff is quite distracting. I thought I'd recommend you take a look at XMLBeans as an alternative to JAXB. It's stable, and it rocks. If you want to take a risk and use XPath 2.0 (I doubt it's a big risk), take a look at Saxon for a great implementation.

  Posted by: jonmountjoy on January 23, 2005 at 02:56 AM

• Hi Jon,

  XMLBeans was a good suggestion. However, we found a few problems with it.

  First, it doesn't give us a way to preserve the reference graph beyond full containment and list order. We'd have to write some caching scheme that resolves any IDREFs. A small but shaggy yak.

  Second, I want to avoid having a separate layer of DAO all together if possible. We're going to be changing the data structures all the time (because we only have a vague idea of what we'll be doing with them). The team didn't have enough patience for a JaxB 1.0/.xsd approach and wants something better. "I'm building a data access object layer for XMLBeans" is not a great answer for, "How's the new scheduling algorithm coming?" Big fat hairy yak.

  The appeal of Xstream is that the team can work with the Java objects they prefer; the cost of mutating them is just breaking some test cases. (Might be able to fix that with xslt.) Scaling it up to handle interlinked xml documents in an xml:db would be some work, but looks straight forward.

  Saxon gives me XPath 2.0, but it's not integrated with anything. To use .xsd substitution groups to get inheritance in XPath, I need the 2.0 spec tied in with the persistence mechanism. Maybe needed. Definitely more work not in focused search algorithms.

  Thanks again,

  Dave

  Posted by: dwalend on January 23, 2005 at 07:15 AM

• If your goal is a persistence mechanism, then look to a persistence technology. It's not clear why you need an XML database ... unless the data you will be manipulating is document-oriented to begin with.

  It seems as if you need a persistence mechanism that will let you start persisting things with only a minor amount of work - maybe even provide an abstraction that allows you to defer the question of which underlying database to use to a later date.

  Since you are already thinking XML, I will recommend a product first, and then justify why your investment in that product will not be mis-spent. Take a look at Xcalia LiDO. It is just one instance of a new class of products which provide persistence based on what have been termed 'plain old Java objects'. I recommend Xcalia because they target XML databases, and I know them to produce decent software.

  Happily, your experience with Xcalia will give you experience with around twenty other products on the market which adhere to the Java Data Objects standard for transparent object persistence. Your code will run un-modified (just some external mappings and factory-properties will change) even if you change database or JDO vendor

  Now the bulk of the JDO vendors (including Xcalia) target relational databases or object-databases. And this is something you'll want to consider. You seem to want to use XML for its low level of commitment to a given data-structure, but surely SQL is just as useful as XSL for evolvability that way? It may be a pain stuffing an object graph into the relational paradigm, but it's an even bigger one sticking it into a hierarchical paradigm! Not that you'll care much using JDO, except when you have to synch your already-persistent data with the latest rev of the object model ... and even then, most vendors provide tools that write the SQL (both initial and all updates) for you!

  Having been a user of JDO for over a year now, I can truly say that it helps you to forget about the persistence 'yak' and concentrate on the real problem at hand. In the relational space I use MVCsoft's implementation, which has a nice homey fit (and price) for my needs as a solo developer. For use in an enterprise situation (for example, where you farmed out the SQL mapping to someone who knew databases better than you), you might choose a bigger vendor. Xcalia's market differentiation is that you can mix and match quite a range of persistence targets - not just relational databases.

  A good JDO resource is JDOCentral.

  Posted by: dtbullock on January 23, 2005 at 05:19 PM

• Yak-shaving seems more of a displacement activity for someone without a real job to do. A writer-in-residence who spends all day sharpening pencils is not the same as a lumberjack who takes a fews days off for tool honing.

  Posted by: arae on January 24, 2005 at 05:30 AM

• arae,

  It's not so much that the job isn't real. It's more that the job is big and long term, and the would-be yak shaver has a free hand to decide how to do it. A writer-in-residence who spends a week learning to sing Russian folk songs when he's got a novel due in two years. Yak shaving seems to have something to do with explaining why you're doing what you're doing in the near term to reach a far term goal.

  dtbullock,

  Thanks for the pointer to Xcalia. I haven't got to pull out my JDO hammer in about two and a half years. JDO over XML is pretty close to what XStream gives me. (I'd point to my early, early FOSS JDO precursor, but some things are better left behind.)

  Dave

  Posted by: dwalend on January 24, 2005 at 07:25 AM

• Since we don't know what sorts of data structures we'll need, but do need to manage a lot of data early, XML:DB and xml binding technology looks like a good fit. Have you thought of using Javaspaces, and throw your data structures in as Entries? --Calum

  Posted by: calum on January 25, 2005 at 05:40 AM

• Yup, XStream will let you persist your entire object graph all at once. JDO (more of a 'way' than a hammer, IMHO) would:

  • allow you to have only parts of your graph in memory at the one time
  • provide a query language that fits naturally with a Java object model
  • provide transactions should you need them

  You still haven't justified why XML deserves to be part of this picture?

  Posted by: dtbullock on January 27, 2005 at 08:36 PM

• calum,

  Thanks for the suggestion. I'll have a look. We've talked about a blackboard-style system for keeping up with what's going on.

  dtbullock,

  Aha -- More details: XML's already in the picture. The inbound details and outbound details are XML. One of the receiving ends is a fairly limited device that already has to have core XML libraries on board, so a large part of the XML tax is already paid.

  It's got some benefits, too. Generating test cases and pretty pictures will be cheap. The management's bought in to it heavily to support long-term interroperability. etc. They'll see relational database technology as a yak.

  But you're right. The API and thought patterns I ultimately want to use line up well with JDO. I seem to fall back into the problem of a first class/second class object split no matter what.

  Thanks,

  Dave

  Posted by: dwalend on January 28, 2005 at 08:54 AM

This work is licensed under a Creative Commons License.