Fabrizio Giudici's Blog

And I made it

Now I can say, at last, that I'm a JavaOne speaker, really, not only a successful submitter. This year no headache, sore throat, cold, flu, pain in the ass prevented me from performing my talk. Not even the stinking MacBook Pro power supply that broke again a couple of days ago (men, I had to buy the third in two years). There could have been more attendees to my talk, but an agenda change a few days ago anticipated Romain's and Chet's talk to my same time slot, so figure it out. But it's good as it is.

Whoa. I presume I'll have an adrenaline drop, now.

PS Yesterday and today I met other people too, but since I didn't keep the list I'm surely forgotting somebody: Adam Bien, Michael Bien, Chris Adamson, Joshua Marinacci. I also met again Dennis Reedy after a few years. Unfortunately I haven't met Kohsuke so far, so I couldn't offer him the promised beer, and Romain is up there busy in signing copies of his book. But there's one day more.

PS2 wsnyder6, I owe you a beer too. If you are around, come on.

Community One

Well, I really can't express all my feelings in these days - even because I'm still suffering from the jet lag. My first thought is that I'm happy to be in San Francisco again, after six years, and to attend Java One again. I didn't remember the Moscone Center was so huge and yesterday morning it took some time for me to find the room for the NetBeans track.

As I said so many times, in this age the real added value of conferences is to breathe the air and to personally meet people. The latter point is especially true when it happens that you "know" people for a long time, but only by means of virtual channels such as the email (I'm not talking about other "social" tools because I don't like them).

So, I must say that my first great joy for the Community One day has been to meet personally Geertjan Wielenga (which is the main responsible for me being part of the NetBeans community) and Tom Wheeler, from the NetBeans Dream Team. I've also met Glenn Holmer, which brings to three the number of NBDTers I've met so far (the first one being Vincent Brabant met at JavaPolis). Today should be the turn of Adam Bien.

But yesterday I've met so many other persons. Here's just a partial list, in random order:

Judith Lilienfeld
Tim Boudreau
Tomas Pavek
Geertjan Wielenga
Arseniy Kusnezov
Yarda Tulach
Rich Unger (I'm particularly happy of meeting him since he was recently in Italy but we weren't able to be in the same town at the same date)
Adam Myatt
David Botteril
Chris Palmer
Tor Norbye
Tinu Awopetu
Ruth Kusterer
Janice Campbell
Fabiane Bizinella Nardon

I've even shot some photos, but now I can't find the time for uploading them.

And this was just "JavaOne Day -1"...

From a technical point of view, I'd say these are the three main points I've seen so far:

The official release of OpenSolaris Indiana. Roman just blogged about having installed it in place of Windows on his laptop. So maybe I'll be finally able to install it on my MacBook Pro? But not before Thursday, it would be embarassing to erroneously scratch a partition before my presentation... :-)
The support for PHP in NetBeans.
The support for JavaScript (including a debugger) in NetBeans. This has been probably around for a while but I didn't noticed it.

Latest refinements

Well, tomorrow morning I'm leaving for San Francisco. There are two basic reasons for my excitement:

The last time I've been in San Francisco was in 2002 - a lot of time ago.
I should have been there in 2005, since my paper about Jini in Formula One had been accepted; but I fell ill the day before leaving. The presentation was performed by two excellent co-workers of mine, but my morale had been under the ground for weeks. So this is my second chance.

There will be intense days next week, also because I'm trying to release a new stable version of blueMarine and there are still failing tests. On the other hand, no problems of course for the JavaOne demo - I'm just trying to see whether I manage in integrating the latest release (0.5.0) of NASA World Wind, which is still hot (published two days ago). Thanks to Dave Collins from the WWJ forum that sent me a patch I've been able to deal with a few API changes and blueMarine seems to work. Now it's a matter of passing some more integration tests to see whether the update qualifies for the demo.

I'm pretty excited for this release since it includes an OpenStreetMap layer, which is pretty useful - and looks nice, as you can see (Pat Murris also blogged about it).

A little desktop quiz

This blueMarine screenshot has been just taken on Mac OS X. Do you note something different than usual? Clicking the image you can see the original capture. I'll pay a beer at JavaOne to the first guy that guesses what's going on (*).

(*) This is the second beer I'll have to offer, the first being due to Kohsuke.

Apple's Java 6 on Mac OS X available

Now the scoop is not that we had to wait 1.5 years before it to be available, but the fact that it only supports 64bit Intel processors. No support for 32bit, no support for PPC. Yeah, PPC is dead, but how many existing installations exist with PPC and 32 bit Intel? And how long you'll have to wait before there's a decent percentage of 64bit installations so you can put it as a requirement for your app? Not counting that if you bought a Mac earlier than 2 years ago, like me, you just need to buy some new gear to start developing with it. Of course this is just whining. As a lot of people will hurry to say today, life is good, Apple is nice, and there's absolutely no need to worry about.

Benchmarks and (less) surprises (?)

As a follow-up of my previous post, I've cleaned up my benchmark code (the log files have been reduced in size from 40MB to 800kB, some bugs have been fixed and the UUID strategy has been chosen for generating primary keys in the database, thus reducing contention chances with multiple threads).

At this point the impact of logging should be neglectable, thus the comparison makes more sense (if you haven't read my previous post, the test reads metadata from about 150+ photos and insert them into a Derby database, with both single- and multi-thread approaches; the test is run with different software combinations but always on the same hardware, my MacBook Pro first generation, 2GHz, 2GB RAM; numbers are seconds per photo, the lower the better).

Workers Linux Ubuntu 8.04               Windows XP                      Mac OS X 10.5
        Java 1.6.0_06   Java 1.5.0_15   Java 1.6.0_05   Java 1.5.0_15   SoyLatte 1.0.2  Java 1.5.0_13
-----------------------------------------------------------------------------------------------------
1       0.50/0.35       0.55/0.47       0.48/0.37       0.52/0.47       0.80/0.66       0.80/0.74
2       0.40/0.26       0.42/0.42       0.32/0.25       0.34/0.33       0.49/0.43       0.47/0.50
3       0.39/0.26       0.41/0.42       0.32/0.25       0.43/0.39       0.48/0.43       0.52/0.53
4       0.39/0.25       0.41/0.45       0.31/0.26       0.45/0.41       0.49/0.43       0.53/0.55

In this round of tests I introduced a double benchmarking for every platform, by using both the "Client" JIT (java -client) and the "Server" one (java -server). Here are some points:

I discovered my ignorance in a basic aspect of the configuration of the VM. While I presumed that by default the "Client" JIT was always selected, I learned that this isn't necessarily true since Java 5. This document clearly states that when running in machines defined as "server-class" (at least 2 processors/cores and 2GB of RAM, and my MacBook Pro falls within this class) the default JIT is the Server one.
In spite of that, the Server JIT is selected by default only on Linux. Window XP and Mac OS X still select the "Client" one. While we are somewhat used to Apple's policy of doing things differently, Windows is a bit of a surprise because it's another Sun-made VM.
This explains a big deal of the differences seen in my previous round of benchmarks: Linux resulted the winner since it was the only one running with the Server JIT, which is clearly the best option since the benchmark doesn not use any guy, but does repetitive tasks.

So at least part of the surprise in the previous set of numbers has gone. But several new questions arise:

Comparing Java 5 VMs omogeneously (Client JIT vs Client JIT and Server JIT vs server JIT) Mac OS X still has the worse performance.
Turning the Server JIT on Mac OS X and other operating systems does almost no good: small improvements with a single processor, and no improvements at all with multiple processors (indeed, it also looks like there's a slight degradation of performance in some cases).
After the verbose logging went off, SoyLatte Client JIT is no better than Apple's JVM. Server JIT for SoyLatte is marginally better, but much slower than Linux or Windows.

At first sight, one could argue that the Server JIT does not bring advantages on any operating system, but the bad performance of SoyLatte is raising doubts about Mac OS X. Is there anything specific in Mac OS X that makes things go worse (again, maybe the file system performance)? Or, given that SoyLatte is still young, a lot of optimizations have been disabled?

The instructions to reproduce this are in my previous post, this time you must use -r 5368 for checking out the blueMarine code.

In any case, at least with Windows and Linux I'm reaching my performance goal, which is 0.20 seconds per photo. Indeed, considering some known issues of my code, I should be able to make better than it.
*** Edited to add:

To make things easier to read, I'm adding a table with normalized values. The percentages indicates how slow a test runs (e.g. 100% means that it runs at half speed). Best performers are marked in green, worse performes in red; also a specific comparison among Java 5 only VMs is given. In each column pair, the "Client" JIT figure is at the left side, the "Server" JIT figure at the right.

Benchmarks and surprises

I'm posting some results from my latest benchmarks on the Metadata facility of blueMarine - take them with half a pinch of salt, since I've checked them but not double-checked, sorry but I have just a little time in these days. They gave some little surprise to me.

In short, the test takes about 160 photos and imports all the metadata into a Derby database (details for reproducing it below). In total, 10000+ records are imported. The tests run with a classic Master / Worker pool with 1, 2, 3 and 4 workers on a MacBook Pro 2GHz, 2GB RAM in all the combinations of (Java 5, Java 6) x (Mac OS X 10.5, Linux Ubuntu 8.04, Windows XP) - note that I've used Soylatte on Java 6 since I don't have a 64 bit computer and can't run the Java 6 previews from Apple (thanks, Apple). The benchmark figure is the number of seconds needed to import a photo, so the lower the number, the better.

Linux		Windows XP		Mac OS X 10.5
Java 6	Java 5	Java 6	Java 5	Java 6 (*)	Java 5
0.48	0.73	0.60	0.70	0.83	1.00
0.37	0.58	0.39	0.46	0.56	0.63
0.38 - 1.13	0.58	0.39	0.52	0.60	0.68 - 1.04
0.36 - 0.76	0.60	0.48 - 0.78	0.56	0.61	1.06

(*) SoyLatte 1.0.2

Before commenting, keep in mind the following points:

The tests still run with logs at maximum power (each run produces 40MB of files), so the computation is more disk intensive than it should
There are some optimizations I have to do yet, for instance photos are read multiple times
Tests with multiple workers have a bug so about 1% of data don't get imported (there must be a failed transaction somewhere and it doesn't get properly logged), but this low number of errors can't change numbers dramatically.

So I expect to have even lower numbers after some fixes, but the comparison is fair across all the environments. Here my points:

As expected, there's no advantage in having more workers than available processors (2). In this case, tests have a huge variance, since some times they trigger a lot of contentions in the database (with a lock timing out for one second and causing the transaction to be re-run).
Java 6 is really faster than Java 5 and for free.
Mac OS X is clearly the loser here, even in comparison with Windows XP, especially with a single processor. Even Java 6 is not exceptional, but I presume the problem is related to the low performance of the file system, that I've already measured some time ago.

Now I've got a simple question. Unfortunately I don't have a quad-core, so I can't test: is it expected that Derby scales well with four workers? Or should I put a limit to 2? When contentions on the database don't occur, it seems that even with only 2 processors a larger number of workers don't introduce significant degradation, which is promising; on the other hand, when contentions occur things are really worse. And the more workers, the more chances of having a contention.

To reproduce:

Check out sources with svn co https://bluemarine.dev.java.net/svn/bluemarine/trunk/src/blueMarine-core -r 5237
Check out test files with svn co https://imaging.dev.java.net/svn/imaging/trunk/www/TestSets -r 49
Open the project blueMarine-core with NetBeans 6.0.1 or NetBeans 6.1
Quit NetBeans just after opening the project
Run from the command line ant generate-platform nbms
Edit Metadata/MetadataOperations/nbproject/private/private.properties and add the line test-unit-sys-prop.testset.folder=path-to-TestSets
From the command line, go to Metadata/MetadataOperations
run ant test |& grep seconds

Historic series of profiling data

After fighting with some showstoppers in the NetBeans Profiler (involving RCP projects) and finding a decent workaround, I've started the tuning of the Metadata facility for blueMarine. I've already done tuning in the past, of course, but I have always had some frustration in how I easily lost the traceability of the thing. I'm giving an example: a few days a go I ran the profiler and got some figures; in particular two methods were the hot spots of the test and I realized it was not good. Working on those methods, I was able to push them down in the list. To help during the job, I exported some dumps of the NetBeans Profiler, manually spotted the most important numbers and manually tracked them build after build.

Now the hot spots distribution is better, but there's still work to do and I won't be able to work on it before some days. When I'll put some further optimization, I will get a new list of hot spots that will look better. And so on until I feel satisfied for this interaction. Then I'll work for several weeks again on new features and bug fixing, leaving tuning alone. I'll run another tuning session probably in a couple of months, etc. By that time, it could happen that the design changes have at least partially invalidated the optimizations done in the current tuning session. Looking back at the code I wrote years ago (and some code I wrote for customers) I can see a lot of maybe-smart optimizations that now could be pretty useless, since the context where they were good has changed (often I see in customers code that this is even made worse by premature optimization, which greatly increases the chances that the selected optimizations are inappropriate).

Of course, you can deal with it: just run another session of profiling, get the new list of hot spots, etc... The point is that since I run profiling sessions every in a while, I'm likely to have forgotten a bit about the context. I can fix this by writing comments and entries in the issue tracker, but this is time expensive. My point is that it would be nice to put some automatic stuff in the CI facility. For instance, I could run a test in profiling mode, dump the hot spots data and plot a graph showing the values (percentage and time ticks) for the five methods on the top of the list and a sixth record with the data of all the rest. This would save me a lot of time and - in the spirit of CI - it would be done incrementally for each build, allowing you to relate repository modifications with sudden changes in the performance (e.g. you see that a method at a certain build has suddenly moved up to the top of the hot spots, and you could easily track the commit version and find out which changes are responsible for that.

What do you think? I don't think there's a plugin for Hudson already available, but I could make one - I think there's already a plugin for plotting graphs if you produce a file with a given format, the only thing I need is a (FLOSS) library that allows me to read a profiler dump.

The troubles with dinosaur and close technologies...

Adobe has just announced that it will be impossible to have the next Photoshop to support 64 bits on Mac OS X systems. The reason is that Apple notified, the past June, that it won't support 64 bits for Carbon, so Adobe has to rewrite it on Cocoa and this would affect about one million (!) LOCs.

While in the past Java has had big troubles for supporting the desktop, and nowadays still a few issues remain, it appears clear how big is its advantage in such scenarios: with Java you can get 64 bit support almost for free (supposing you have a good 64 bit VM support, which happens in Windows and Linux).

It's also appalling to me to learn that strategic software manufacturers such as Adobe (I don't think Apple would have been so successful in some marketing segments that were strategic in the past without the contributions of superb products such as the Adobe Suite) don't get early warnings from Apple about its close technologies such Carbon and Cocoa (Adobe's statement refers to WWDC news). This makes me understand even better how important is to work with open technologies.

blueMarine goes semantic

Today Jazoon '08 has published the final program and I don't see my proposal there, so I've not been selected. Usual business in the conference world, sometimes you get accepted and sometimes not - it's only strange that I didn't receive any personal notification about that, it must have been lost somewhere.

In any case, the proposed paper was named "blueMarine goes semantic" and - you get it - it was about the integration of blueMarine with semantic technologies - note that I'm not strictly referring to semantic web, but semantic technologies in general. There is some preliminary code (not yet in the public source repository) and, given the Jazoon news, at this point I'm suspending the work until past JavaOne. So, this is just a teaser, expect me blogging about that after next May.

PS And not, this is not another April's fool :-)

I must admit: Eclipse is the best tool in its class

Well, this is the sound opinion of a person who has been using it for years. There's no history: Eclipse is just the best tool of its class. I've tried some alternatives, but nothing allowed me to operate quickly and having such a clean result as Eclipse did. Just nothing. And if you're in pain, trying to deliver when operating in the field, maybe in a problematic environment, Eclipse just gets you out of your troubles in a few minutes.

Don't trust of what competitors say, it's marketing hype, Eclipse is just the best tool of its class.

Long life to Eclipse! :-)

Designing flexible method calls

In the latest months I've not blogged a lot about technical topics, yet I'm still intensively working of stuff about my opensource projects (well, of course I also work on other stuff, but I can't blog on that because of NDAs with customers).

The point is that 95% of the development of blueMarine has been focused, since the latest XMas, on the Metadata component. It is a thing to import, store and query metadata extracted from photos and other documents, and I put some demanding requirements on it, including the capability of being pretty modular and extensible. It turned out to be a meta-persistence facility on its own for JavaBeans, that will be even useful for futher developments. Of course I didn't reinvent the wheel, the thing still uses JPA + Derby, but adds a layer of services that are focused on the special kind of JavaBeans I'll be using.

The design has been changed two or three times so far, and I'm pretty happy this is the first time I went in mostly-TDD mode for blueMarine (thus having 90%+ test coverage since the beginning), since I was able to perform dramatic changes with no fear of breaking anything. After some integration tests passed yesterday, I think that the design is almost complete, so I started consolidating and documenting (and also removing some possible over-engineering).

So here it is a quick technical post about it and I like to have some feedback before I freeze the APIs.

As I said, the facility should be extensible, and this implies having stable APIs that can be used by third parties. Stable APIs must be however extensible as far as it's possible. A Metadata interface is central in the facility, as it manages a collection of metadata items associated to a given object. Among other operations, Metadata exposes a findOrCreateItem() and a storeItem(). The implementation is SPI-based, that is independent implementations of providers will be activated on request. Providers might be able to read data from a photo file, read or store from/to an XMP file or work directly with the database. I can't predict in advance which kind of providers will be able in future, including some hopefully provided by third parties, so I'd like to have an open door so the application can pass some special parameter to them.

So, everything boils down in how to construct a method whose arguments are flexible enough, yet controllable, and also keeping an eye on readability of the code. Up to a few months ago, my usual way was to avoid fixed parameters such as:

public void myOperation (int param1, String param2, float param3);

in favour of the use of a specific JavaBean and a method with a single parameter:

public class MyOperationParameters
{
public void setParam1 (int v)...
public int getParam1()...
}

MyOperationParameters p = new MyOperationParameters();
p.setParam1(...);
public void myOperation (p);

In this way you can add new properties to MyOperationParameters and make sure that they have the proper default values. Not bad, but it make it worse the readability of the caller code. Of course, you could initialize MyOperationParameters properties in the constructor and/or use a fluent interface, etc... But even worse, from the design point of view, is that MyOperationParameters becomes a focal point and any eventual extension (e.g. adding new parameters) performed by added code must use subclassing, which is evil in this case.

This time I opted for a different solution. For instance, the method storeItem() is declared as:

public int storeItem (MetadataItemHolder holder, Metadata.StoreOption... options);

where holder is a fixed and mandatory parameter; all the remaining stuff is a varargs of classes implementing StoreOption, that are straigthly passed to SPI implementations. StoreOption is just a tagging interface, that is it's empty, only used to have a minimal syntax cheking from the compiler (so I can't pass arbitrary objects). Then I have some concrete classes:

public interface Metadata
{
    public static interface StoreOption
      {
      }

    public static enum StorageType implements StoreOption
      {
        ANY_TYPE
          {
            public boolean includes (final StorageType sourceType)
              {
                return true;
              }
          },

        INTERNAL
          {
            public boolean includes (final StorageType sourceType)
              {
                return sourceType == INTERNAL;
              }
          },

        EXTERNAL
          {
            public boolean includes (final StorageType sourceType)
              {
                return sourceType == EXTERNAL;
              }
          };

        public abstract boolean includes (final StorageType sourceType);
      }

    public static enum ReplaceOption implements Metadata.StoreOption
      {
        DONT_REPLACE,
        REPLACE
      }

    public static class OriginFilter implements StoreOption
      {
        private final String name;

        public OriginFilter (final String name)
          {
            this.name = name;
          }

        public String getName()
          {
            return name;
          }
      }
...
}

As you can see, some of the classes have their own logic, as for the includes() method in StorageType (in contrast, adding several specific pieces of logic to a central MyOperationParameters would quickly turn it into a "small" fat class, if you pardon the pun).

With the proper static import, I can call:

metadata.storeItem(holder, INTERNAL, new OriginFilter("Database"), REPLACE);

which seems pretty readable and self-documenting to me. New SPI implementations can define their own parameter types by just implementing StoreOption. Method implementations retrieve the arguments by a special lookup static method such as findOption() in the example below (which also provides the default value, just to avoid extra checks for null values):

    public void storeMetadataItem (final MetadataItemHolder holder, final StoreOption... options)
      throws Exception
      {
        logger.fine(String.format("storeMetadataItem(%s, %s)", holder, Arrays.toString(options)));
        ReplaceOption scratchOption = MetadataSpiUtils.findOption(options, DONT_REPLACE, ReplaceOption.class);

...
}

Last but not least, using Arrays.toString() it's easy to produce a pretty good logging information such as:

12:04:39.736 [main] FINE MetadataItemProviderSuppor - storeItem(PhotoDataObject[V7210188.JPG], [Basic@131], [INTERNAL, REPLACE])

I don't think it's new stuff, and I'm sure people is already using this style somewhere, but I could not find examples.

So, what do you think?

Mailing list di NetBeans in Italiano

Well, as an exceptional case I'm going to blog here in italian. The point is that we have an italian mailing list for NetBeans, so it's a pretty local interest.

Dunque, o voi tutti utilizzatori italiani di NetBeans, abbiamo finalmente una mailing list in cui possiamo scriverci nella nostra lingua madre. Questo strumento mi è stato chiesto da alcuni di voi durante i NetBeans Days tenuti nello scorso Settembre a Roma, Milano e Cagliari e Sun Microsystems è stata molto gentile nel risponderci positivamente. Janice Campbell, responsabile del programma di localizzazione, ci ricorda che questa è l'ottava mailing list su NetBeans non in inglese.

L'indirizzo della mailing list è nbdiscuss_it@netbeans.org e per iscriversi basta mandare un messaggio vuoto a nbdiscuss_it-subscribe@netbeans.org. Gli archivi sono disponibili su http://www.netbeans.org/servlets/SummarizeList?listName=nbdiscuss_it

Ora, non ci resta che far partire un po' di discussioni interessanti!

A success for the "IDEs of March"

A very smart comment by JavaLobby played with words and called "the IDEs of March" (*) the event that saw two italian JUGs to organize a new edition of the IDE shootout initiated by JUG Cologne. Now that the event is over, I can say that it has been a success: 200 persons could enjoy Roman Strobl (Sun Microsystems), Paolo Ramasso (Oracle) and Vaclav Pech (JetBrains) demonstrating the virtues of their products. A special thanks to our guests that made the event possible (too bad for the problems that prevented an Eclipse speaker to come). We know that attendees really enjoyed the event, so I can only recommend other JUG leaders in the world to repeat it in their home countries.

My congratulations to JUG Genova and JUG Roma, especially to the former since they resumed the activities just one year ago, after a period of stall. Here are some photos of the event (and - as some photos demonstrate - in the better italian tradition the thing ended up in with a great dinner!). All the official photos are available on http://www.ideday.org/index.php/Site/Photos.

(*) "The Ides of March is the name of the date 15 March in the Roman calendar. The term ides was used for the 15th day of the months of March, May, July, and October, and the 13th day of the other eight months. In modern times, the term Ides of March (Latin: Idus Martiae) is best known as the date that Julius Caesar was assassinated, in 44 BC, the story of which was famously retold in William Shakespeare's play Julius Caesar.

Feedback from the IDE Day in Genoa

A quick feedback from the IDE Day in Genoa, waiting for the other IDE Day in Rome to be over before posting a final report. Well, the day in Genoa was a success: while I don't have gotten yet the official data, there should have been 130+ attendees, which is very good. The talks by our guests from IntelliJ, Oracle and Sun were of very high quality, as expected; and the large number of questions proved that they caught the attention of people. Congratulations to the JUG steering commitee for their first large event, even considering that they resumed the JUG almost from zero about one year ago.

In spite of being my home town, unfortunately I wasn't able to attend it because of an engagement in Milan; I only got in in the latest minutes of the panel session. But I fully enjoyed the final dinner based on genoese cuisine and fish... :-)