A few months ago I blogged about how to optimize a large NetBeans RCP project by splitting it in parts and taking advantage of a technique known as "suite chaining". Now I'm going forward as I have some more complex requirements to meet and suite chaining doesn't fit with them.

If you are not comfortable with the layout of NetBeans RCP projects and the related jargon I suggest you to read first my previous blog (at least the introductory part to understand the basic concepts such as "platform", "suite", "cluster", "NBM file").

And now let's recall what I wrote in the previous post. At that time, I basically split blueMarine in three parts (and defined some APIs so people could develop plugins to extend the application):

• blueMarine Libraries. It's a "fake" component which only contains some module wrappers for a certain number of libraries required by the project(s) just for saving some compilation time.
• blueMarine Core. It's the basic imaging components of blueMarine. It can be used by developers for developing other imaging applications.
• blueMarine. It's the actual application.
• Generic Plugin. It's a generic plugin that people can develop for blueMarine.

As you can see a simple, linear chain of dependences emerges from the design and fits well with the "platform chain" concept of NetBeans RCP: basically each project builds upon the previous stage of the chain and delivers a new platform enhanced with new features.



With the latest refactoring, two new components appeared:

• OpenBlueSky. It's a set of components that enhance the NetBeans RCP. It is a new spin off of blueMarine Core.
  
• ForceTen. It's a set of components for showing 2D and 3D maps. It is a new spin off of blueMarine.

The basic concept is that OpenBlueSky must be reusable by other programmers outside the blueMarine ecosystem, as well as ForceTen that also originates a standalone application (and thus uses OpenBlueSky too). These new requirements lead to a more complex set of dependencies, where linear suite chaining can't be used any longer.

Also consider that the previous diagram has been simplified: we should take into account that, for instance, there are multiple dependencies on JOGL (they are best shown in the next diagram).

This lead me to have a sort of Copernican shift. In the previous approach, the final deliverable of any component was a public platform, exposed to further projects in the chain. Now I focus on delivering a set of NBM files (the pink component in each package in the diagram below), much like a standard Java application produces some JAR files, and each project defines its own "private" platform (named "***-base") which is "artificially" created by taking the required NBM files from multiple sources and unpacking them. Each set of NBM files creates a different cluster in the platform. Platforms are now "private" since they are only used by the project they belong to and not exposed to others.

You see that there's an exception, since blueMarine Core and blueMarine are platforms, yet still publicly exposed. This is because they can be used by other people (for instance during the development of a Plugin) and the process of initially creating a platform requires some manual steps, which I want to avoid at the moment. But I think I'll work on it, so at a certain point I'll be able to deliver only a set of NBM files also in these cases (see the final remarks at the end of this post).

Automating the process

Now, a practical problem is to keep everything up-to-date. For instance, let's suppose I make some changes in OpenBlueSky. This means that one or more NBMs are updated and, following the dependency chain, you see that the "blueMarine Core base", "ForceTen base" and "blueMarine base" platforms need to be updated (as well as some further dependencies).

We have some things to do:

1. Update NBMs in chain, which can't be just achieved by copying files, since e.g. a certain module could be deleted or renamed at a point; this means that the old file should be removed from dependent platforms as well.
2. Create / update the platforms with their clusters. NetBeans isn't able to directly compile against NBM files, which are just a convenient way to pack things. Instead, they are expanded following a well known file tree into a platform, which is the artifact NetBeans is able to compile against.
3. Last but not least, you might want to have platforms and NBM files checked into Subversion. This approach not only makes it easier to work on a single sub-project without downloading all the sources of ancestor projects, but also delivers reproducibility as each revision has a well established set of both sources and libraries. Not only you should use Subversion appropriately whenever a certain file has been added, modified or removed, but you have to cope with a subtle problem: since NBM files contain the compilation timestamp, if you rebuild them even with no change in the sources, you always create a file with some changed bits. This means that any time you recompile NBMs Subversion has to commit all of them. Not only this eats up a lot of space in the source repository, but makes it much harder to track changes by looking at the commit history.
   

For dealing with all these problems I've created a new ant task, named "copyUpdatedStuff" (I might change its name as soon as I find something more appropriate). This task takes three arguments:

1. sourceFolders, a colon-separated list of directories where there are some "source" NBMs to get.
2. nbmFolder, a single directory where the NBM files are copied. This directory is kept as a reference to understand which files have been changed since the last time. To solve the timestamp problem cited above, copyUpdatedStuff doesn't compare contents, but the embedded version tag from the manifest of each module.
   
3. platformFolders, a colon-separated list of directories where copyUpdatedStuff expands the NBM files to create clusters.
   

To understand better how the thing works, let's look at how to manage the platform "blueMarine base". First we define the set of directories where to search for NBM files (look at the previous diagram, there are five dependencies):

    <property name="source.nbm.path" value="${openbluesky.dir}/build/updates:
                                            ${bluemarine-libraries.dir}/build/updates:
                                            ${bluemarine-core.dir}/build/updates:
                                            ${jogl-libraries.dir}/build/updates:
                                            ${forceten.dir}/build/updates"/>

Basically we're looking at the build/updates directory of each ancestor project, where NetBeans placed the NBM files after compiling the sources.

Then we define the target directories:

    <property name="nbm.dir" value="lib/nbm" />
    <property name="platform.dir" value="lib/platform"/>

and specifically a set of cluster names to match the ancestor projects:

    <property name="platform.path" value="${platform.dir}/openbluesky:
                                          ${platform.dir}/bluemarine_libraries: 
                                          ${platform.dir}/bluemarine_core:
                                          ${platform.dir}/jogl-libraries:
                                          ${platform.dir}/forceten"/>

And now what we need is just to run ant prepare-platform whenever we want to bring into blueMarine the last updates from the ancestor projects:

    <target name="prepare-platform">
        <copyUpdatedStuff sourceFolders="${source.nbm.path}"
                          nbmFolder="${nbm.dir}"
                          platformFolders="${platform.path}"/>
    </target>

The set of files produced by this task is the following:

   ./nbm
   ./lib/platform/bluemarine_core/config/Modules
   ./lib/platform/bluemarine_core/config
   ./lib/platform/bluemarine_core/core/locale
   ./lib/platform/bluemarine_core/core
   ./lib/platform/bluemarine_core/modules/locale
   ./lib/platform/bluemarine_core/modules
   ./lib/platform/bluemarine_core/update_tracking
   ./lib/platform/bluemarine_core
   ./lib/platform/bluemarine_libraries/config/Modules
   ./lib/platform/bluemarine_libraries/config
   ./lib/platform/bluemarine_libraries/modules/ext
   ./lib/platform/bluemarine_libraries/modules
   ./lib/platform/bluemarine_libraries/update_tracking
   ./lib/platform/bluemarine_libraries
   ./lib/platform/forceten/config/Modules
   ./lib/platform/forceten/config
   ./lib/platform/forceten/modules/ext
   ./lib/platform/forceten/modules
   ./lib/platform/forceten/update_tracking
   ./lib/platform/forceten
   ./lib/platform/jogl-libraries/config/Modules
   ./lib/platform/jogl-libraries/config
   ./lib/platform/jogl-libraries/modules/ext
   ./lib/platform/jogl-libraries/modules/lib
   ./lib/platform/jogl-libraries/modules
   ./lib/platform/jogl-libraries/update_tracking
   ./lib/platform/jogl-libraries
   ./lib/platform/openbluesky/config/Modules
   ./lib/platform/openbluesky/config
   ./lib/platform/openbluesky/modules/ext
   ./lib/platform/openbluesky/modules/lib
   ./lib/platform/openbluesky/modules
   ./lib/platform/openbluesky/update_tracking
   ./lib/platform/openbluesky

Indeed blueMarine also depends on some modules from the NetBeans platform:

   ./lib/platform/ide8/config/Modules
   ./lib/platform/ide8/config
   ./lib/platform/ide8/modules/ext
   ./lib/platform/ide8/modules/lib
   ./lib/platform/ide8/modules
   ./lib/platform/ide8/update_tracking
   ./lib/platform/platform7/config/ModuleAutoDeps
   ./lib/platform/platform7/config/Modules
   ./lib/platform/platform7/config
   ./lib/platform/platform7/core
   ./lib/platform/platform7/docs
   ./lib/platform/platform7/lib
   ./lib/platform/platform7/modules/ext
   ./lib/platform/platform7/modules
   ./lib/platform/platform7/update_tracking

but at the moment I don't know how to automatically put them in the target directory - so I had to manually copy them the first time.

The task also prints on the console some relevant information about what's happening:

prepare-platform:
[copyUpdatedStuff] 0 new module(s):      
[copyUpdatedStuff] 2 updated module(s):  it-tidalwave-bluemarine-calendarexplorer (2.2.10) in: bluemarine-core, it-tidalwave-geo-geoexplorer (1.0.6) in: ForceTen
[copyUpdatedStuff] 0 obsolete module(s): 
[copyUpdatedStuff] Copying and updating: it-tidalwave-bluemarine-calendarexplorer.nbm
[copyUpdatedStuff] Copying and updating: it-tidalwave-geo-geoexplorer.nbm
[copyUpdatedStuff] Copying and updating: config/Modules/it-tidalwave-bluemarine-calendarexplorer.xml
[copyUpdatedStuff] Copying and updating: modules/it-tidalwave-bluemarine-calendarexplorer.jar
[copyUpdatedStuff] Copying and updating: update_tracking/it-tidalwave-bluemarine-calendarexplorer.xml
[copyUpdatedStuff] Copying and updating: config/Modules/it-tidalwave-geo-geoexplorer.xml
[copyUpdatedStuff] Copying and updating: modules/it-tidalwave-geo-geoexplorer.jar
[copyUpdatedStuff] Copying and updating: update_tracking/it-tidalwave-geo-geoexplorer.xml

What's next

The relationship among my projects have turned out to be a bit cumbersome, but after all they reflect the dependencies in the new design. And the stuff seems to work. I also like the fact of focusing of the NBM files as a final deliverable for a component since it matches the same approach we have with JAR files. But one of the problems is a sort of redundancy (because of how NetBeans works): we need to keep both the NBM files and the platform folders, even though they share the same contents. Even worse, they both get checked in, thus wasting some space in Subversion. I thought a bit of it, and tried to use copyUpdatedStuff to add a target to build.xml that would recreate the platform starting from the NBM files in case it doesn't exist. In this way one could avoid putting the platform in the repository and the first time a project is checked out, the platform would be generated on the fly.

Unfortunately, at the moment this is impossible: in fact, before being able to run ant on a NetBeans RCP project you must first set the referenced platform (by means of the Platform Manager tool of the IDE), but we're supposing it is not present at the very beginning, since ant should create it! I'll investigate how work around this.

If you want to try it, copyUpdatedStuff sources can be checked out from https://bluemarine.dev.java.net/svn/bluemarine/trunk/src/UpdateManager. Beware, it's still alpha and has got some bugs.

PS I'll blog about OpenBlueSky and ForceTen as soon as I have some basic documentation for them. But the good news is that you can check them out from https://openbluesky.dev.java.net and https://forceten.dev.java.net and try them since now all the dependencies on blueMarine has been removed.



Technorati Tags: NetBeans, NetBeans RCP, suite chaining, blueMarine, ForceTen, OpenBlueSky