Meeraj Kunnumpurath's Blog

Generate or Handcraft?

One irritating thing I used to find in my early years of programming was the amount of time I spent on handcrafting details. Over the years, slowly but steadily, I have learned the art of meta-programming and I would say it is now the best tool in my programming arsenal. So what is meta-programming? It is programming using metadata. And what is metadata? It is data about data. For example, databases have always provided metadata information in terms of dictionaries and meta tables. Database metadata include, information about the tables in the database, indices on tables, procedures that are available etc. Similarly you can have metadata about your application. Whenever I start hand-coding details, I try to extract the details in terms of metadata and write a framework that can interpret the metadata.

One of the main disadvantages in handcrafting details is that details change frequently forcing you to make changes to your programs to capture the changes. The more changes you make manually, the more bugs you introduce in the system. In my experience hand-coded details create the most number of bugs in the system. So, how do you get around this? The answer is metadata!! You externalize the details from your code to metadata. Then have an abstract framework that can interpret the metadata. For this you can have a number of strategies like static code generation based on metadata or a more dynamic reflective framework that can interpret the metadata. This means the only place you introduce bugs is in the framework. A bug in the framework will be manifested in every single artefact of your system that uses the framework, which makes the bugs easier to find and fix.

J2EE introduced the art of meta-programming to the enterprise Java world. With deployment descriptors you externalize a huge amount of meta information from the code and you delegate the tasks of interpreting the meta information to the containers. This was further enhanced by the introduction of great tools like XDoclet that used doclet based metadata to extract more details out of the code. Then came tools like Middlegen that generated your entity beans from the metadata in the database. If I look back to around four years ago on how I wrote entity beans and how I write them now, I would say it has been a quantum leap. This doesn't need to stop there. In my current project we used fastlane readers backed by stored procedures for bulk reads. All our fastlane readers are generated using a custom Ant task and Velocity based templates driven by metadata in the database. The next step would be to capture your domain logic in terms of metadata (you can even have a custom domain language) and drive your business components off that meta information. Less code you write, less bugs you have and more maintainable your system is.

Some of the advantages I have found on using meta-programming are,

• It is easy to find and fix bugs
• You spend less time with details
• Your system is more adaptable to changes
• You spend less time on repetitive and redundant tasks
• Your system is better designed as you spend more time in writing a robust framework and less time with the details

Comments

• Agreed we currently have 116 ejb's in the projects we are working on we are using xdoclet and to generate as much as we can when building the project. Im sure generation has saved many man hours of coding over the duration of the project. ** Disclaimer ** The project im currently working on does happen to be the same project as meeraj

  Posted by: dantheman on September 03, 2003 at 03:50 AM

• Meta-Programming Very well expressed Meeraj, I work with Meerak at EDS in the UK and find that he puts his words into action. I feel that one of Java's sterngth's is it's nature as a general purpose language. I also find that is it's weakness. It can be very hard to express the domain problem as Java. The transition from idea to application is interrupted by swathes of plumbing code relating to the technology being used. Which one would expect in a general purpose language. The value of meta programming as demonstrated by Middlegen/XDoclet and the like is the ability to express the solution you are working on within the context of the problem you are solving. XDoclet solves this with doclet tags, middlegen uses the database combined with a preferences file and GUI. Other solutions can be XML configuration or UML like AndroMDA. I think the important aspect is not the tool but the attitude. It's an ucomprimising stance on DRY (Do not repeat yourself) and KISS (Keep it simple stupid). That is to try to write the minimal amount of code which expresses the problem being solved - in this case by abstracting the 'plumbing' away.

  Posted by: neilellis on September 03, 2003 at 03:59 AM

• Agreed And an untold amount of time was spent initially implementing changes to optimistic locking policies and the like. Now you can apply your policy to all EJBs easily.

  Posted by: neilellis on September 03, 2003 at 04:01 AM

• Meta-data Good article. I am also an EDSer currently working on a J2EE project. We have entity beans for the persistence layer. During the first phase of the project we hand crafted the entity beans but used ejbgen to generate the deployment descriptors (by embedding javadoc comments for ejbgen). Hand crafting the entity beans was laborious as they simply mapped onto the database. For the next phase of the project I wrote my own entity bean/value object generator based on the database metadata. It is very simple and only took a couple of days to code. However, the savings are great as we no longer need to code these classes by hand. The code is not template based but we have the source so it's easy to change. The entity beans are based on some ejb design patterns for primary key generation so we subclass a particular class. I didn't know about middlegen, it looks quite promising. What I have found with commercial code generators in the past is that they don't quite work in the way you wish. Originally we were going to implement the project in .NET and I was surprised that when we switched to Java we couldn't generate the entity beans. Within .NET it is possible to generate "typed datasets" based on the schema. Although my code generator now makes up for this.

  Posted by: jdthompson123 on September 08, 2003 at 05:54 AM

• Generative Programming This has been covered in some texts on domain engineering, such as Generative Programming. Studies of top-notch companies show that building software that creates your code gives huge productivity gains (x5-10). It also requires really tight discipline and (I can hear the XP people cringing) a strong process. You're generally building a software factory to stamp out lots of very similar chunks of code. The main issue is it takes about 3 of those chunks of code to pay back the investment of writing the factory, so getting management to invest in that upfront isn't an easy task. The books/studies are generally geared for creating software for multiple product lines, but the same reasoning applies to a smaller scale where you're creating lots of similar code (like EJB's) for a single product.

  Posted by: ckessel on September 08, 2003 at 11:27 AM

• Adopt metadata and forget CMP I'm another EDSer working as contractor for EDS Brazil. I think adoption of metadata speeds up software development but there are more annoying problems when developing J2EE applications. The need of repetitive deployments into app server just to test little pieces of code probably is the biggest issue. This is due to CMP objects that, of course, require a J2EE container. In a project, we were using WebLogic and I tryed to debug our application running inside WebLogic. But, besides it is possible to do thanks to JPDA support, it is not reasonable because of simply unacceptable deploying time. So, we returned 50 years back in software development and we used stupid "println" to debug code. There are interesting articles about how to decouple entity beans and other EJB patterns, aiming to enable fast testability and debugging out of an J2EE container. Unfortunately the resulting class diagram seems like a spaghetti that ends up covering the original idea. Then I decided to switch to JDO and related technologies... in particular, open standards and freeware tools. I tryed OJB, but it's not very mature, in spite of its features. Finally I adopted Hibernate2 because of its very mature, in spite of some features :-(. Returning to Generate vs Handcraft: Using ArgoUML I can start from requirements and generate UML diagrams. Using AndroMDA I can input UML diagrams and generate database schema, Hibernate XML files and Hibernate Java classes. It is also possible to use AndroMDA to generate other artifacts besides entity classes, such as Control and View layers. Using stereotypes in UML it's possible to generate classes dependent and independent of J2EE container, in cooperating pairs. The J2EE-dependent class will be used to hold transactions inside J2EE container. The J2EE-independent class has the working code we are interested on, the business logic I mean, that can be tested out of J2EE container. One may also use stereotypes to generate JUnit classes, a third cooperating class, that calls methods from J2EE-independent classes. Finally I can debug and test the model and control layers out of an application server because most of generated classes are Plain Old Java Objects. Conclusion: Using code generation and the right development strategy, one may reach excellent levels of productivity. Fernando Gomes mailto:fernando.gomes@bigfoot.com

  Posted by: frgomes on September 08, 2003 at 03:39 PM

• Middlegen and AndroMDA I'm another EDSer working for EDS Brazil. Middlegen is an excellent tool. It takes your schema and generate dozen of files: entity beans (CMP), Hibernate XML files and classes (an excellent alternative to CMP), DDLs to [re]create your schema, JSPs and Struts classes. Remember: you will start from an existing database schema. I also recommend AndroMDA. The main advantage is you can start from UML, maybe using ArgoUML or Poseidon, then generate almost the same Middlegen generates. Starting from UML instead of already existing schema may be a big advantage for new projects. Depending on your needs, you may prefer one or other tool. Both of then uses "plug-ins" to implement specialized generators and Velocity under the covers. Fernando Gomes mailto:fernando.gomes@bigfoot.com

  Posted by: frgomes on September 08, 2003 at 03:55 PM

• Meta-data It seems quite a few people are interested in code generation. I found an interesting article on onjava at the following url. http://www.onjava.com/pub/a/onjava/2003/09/03/generation.html There are some useful references at the end, the writer of the article has written a whole book on the subject.

  Posted by: jdthompson123 on September 09, 2003 at 12:36 AM

• Meta-data not so popular... Yes... Unfortunately few people adopt code generation, in spite the productivity this approach can lead one to. Maybe because the generated code is not througly finished in general. Many companies/developers have their own [customized] code generation suite, composed by tools, templates, libraries, components and so on. It seems that there's still a lack of something. For instance: I'm currently using Poseidon + AndroMDA to generate Hibernate classes, value objects and Strut classes (in spite I hate JSP!). The conclusion is: when I create UML model in Poseidon I have to take care of many details AndroMDA will take a look after. Depending on the UML model, AndroMDA may even generate Java code that does not compile. One may argue that my UML model is badly designed. Maybe, maybe not. There are situations when AndroMDA generates duplicated methods because of multiplicity of dependencies and navigability in the UML model. Well .... I'd like to offer such navigability to my end user's process and AndroMDA should take take of duplicated methods. This is just one example of a such situation when automatic code generation needs (1) additional fix by hand or (2) back to the previous step to avoid badly generated code. Maybe thats why sometimes people prefer to write code by hand. Fernando Gomes mailto:fernando.gomes@bigfoot.com

  Posted by: frgomes on September 11, 2003 at 12:28 PM

• If you need to generate something is often wrong In general if you are generating hundreds of similar classes it means you should have one class that has a bit of intelligence built in (like mapping objects to tables dynamically) rather than one EJB for each table etc. You get better performace as well as more maintainable code.

  Posted by: c_armstrong on September 13, 2003 at 09:18 AM

• If you need to generate something is often wrong Yeah, it either comes down to either having a more generic dynamic approach or generating specific ejb objects for each table. However, if you went for the dynamic approach is that scalable? i.e. it wouldn't be possible to use EJB with the dynamic approach, or would it? I think .NET follows this approach with their DataSet objects, which are dynamic. Late binding does have it's own problems though in that errors will be found at run time rather than compile time. Although the testing should find these problems. The downside to generating the EJB objects is if the underlying tables change. My approach is to make the changes by hand if the table change is minor, otherwise regenerating is quicker. Although table structures do not change so much in practice. I have only been working in J2EE for a year or so after moving from an MS background. There seems to be a whole range of database access technologies e.g. plain old DAO pattern, JDO, EJB etc. it is difficult to see when each is appropriate.

  Posted by: jdthompson123 on September 15, 2003 at 05:31 AM

• Abstraction = obfuscation I disagree, but I realise I'm one of a dying breed :-) The project I'm working on right now uses this abstract framework/metadata approach, and to me it seems like we're using a sledgehammer to crack a nut. This isn't the first time - must be the 3rd or 4th "generic" system I've worked on now (each developed by different companies)... IMHO too much emphasis is placed on creating a generic or customisable "solution". What we invariably end up with is an overblown, slow, impenetrable mass of code. You can't see the wood for the trees when trying to fix bugs. The project runs over, reliability is questionable, and any theoretical advantage "gained" by having a generic solution is lost by the sheer maintainance headache. I'm sorry, I know I'm probably screaming into the face of a tornado, but the Keep It Simple, Stupid principle is far more sensible in my book.

  Posted by: pogo on September 16, 2003 at 06:14 AM

This work is licensed under a Creative Commons License.