Chet Haase's Blog

Persistent Questions

As Kermit once almost said, "it's not easy being free."

Having a free and popular download has its advantages. Developers like the platform, they grab updates, they post bugs, they participate in forums, they come to conferences, they read blogs (I hope), ....

But being free and popular also has a downside (beyond the whole "revenue" thing): we don't necessarily know what parts of our libraries people are actually using.

Sure, we know the big-ticket items. We know the features that are most requested, we know the areas that get heavily used because of constant bug reports and JDC votes, we know the industry-trendy areas of the platform that we need to keep working on, etc.

But what's harder to gauge is the use of the more subtle parts of the platform, APIs that developers probably use a little of for specific pieces of functionality, but that don't get much public fanfare because they're just basic plumbing. I mean, I might mention to someone that I got a new granite countertop in the kitchen, but I probably wouldn't bother talking about having installed a brand new drain in the sink. It's just not very exciting.

And what makes this particularly difficult to gauge in a platform as broad and huge as Java is that there are often many different ways of accomplishing the same task.

Take persistent storage for application preferences, for example; I'm talking about the "my app reads this stuff every time it starts" data, not the data stored with particular documents that may not be read at startup. There are so many ways of accomplishing this rudimentary task that I probably can't think of a majority of them. There's the Preferences API, there's the Properties API, there's roll-your-own file approaches, there's lots of different XML-based approaches, ... and on it goes.

Which ones are developers actually using? Any of these? Some I haven't mentioned? None of the above? All of the above? The sad truth is I don't know. It comes back to the problem I stated above; people download our stuff, write the applications, get it to work, ship it ... and we don't know what they decided to use (unless problems crop up; then we hear about it. A lot).

So now we get to the crux of it, the meat in the puff pastry, the cherry in the sundae, the pebble in the shoe:

• What do you use to persist your application data?

At this point, or maybe way back at the beginning, you might ask "Why does he care? Isn't he a graphics guy?". A fair question and one which, in recent weeks, has been much on my mind. Yes, this is pretty far outside the graphics realm. But one of the hats I wear here has a label that says "member of the startup performance team", and it's in this capacity that I care.

XML is often associated with server-side or web-related development. Its angle brackets are spewed all over the place in those realms. In desktop apps, it may be used as a file format, but there tend to be much bigger issues in the desktop space, so XML doesn't get a lot of press for desktop apps.

Because of this history, performance of XML-related technologies has been more server-side focused: how fast can your business logic crank on some DOM thingie? Issues of startup, like how fast can you parse that XML file, are pretty irrelevant in this space; if it only happens once for a long-running server application, it's not worth worrying about.

But if desktop apps are using these technologies to persist data that's read at startup, then it becomes an interesting area in terms of desktop app startup performance. Server apps may stay up for days or weeks at a time, but desktop apps are lucky if they run all day. In this context, the speed at which those apps start up becomes much more crucial. And if there are common things that all applications are doing at startup time, then we'd better make sure those things are performing up to snuff.

So, how about it:
Can you tell me what to tune here? Should I be benchmarking Preferences? Or Properties? Or one (or all) of the several XML-related APIs? Or are people just rolling their own and we should just make sure file IO is fast enough to handle it? Or is this simply not an issue and I can return to thinking about pixel performance?

Lay down a quick comment below if you have any tips here for me. Sure, it's anecdotal evidence, but anything helps.

Comments

• Hi, Chet; I use Preferences. I also have used Preferences backed by a database (don't know where that fits in your overall survey).

  Cheers,
  Laird

  Posted by: ljnelson on August 19, 2005 at 09:04 AM

• I must admit, the only one I've used is Properties. Basically, I try to avoid XML as often as I can because it always feels like overkill.

  However, one expects that because Properties is relatively simple, that it doesn't need much optimisation. But, who am I to say, I haven't checked. Therefore, I'm sure that Preferences will need your attention more, because it adds more complexity.

  Also, even if Java was not free, I don't see how you would know any better what APIs devs are using.

  Posted by: arooaroo on August 19, 2005 at 09:31 AM

• I use a couple of Properties files to manage my application's preferences. One Property file is for individual preferences and one for more global preferences. Probably keep 20 or so properties in the files.

  Posted by: btschumy on August 19, 2005 at 09:33 AM

• Hi Chet, We use XML Preferences stored in an XML native database for our app.

  Posted by: adrianff on August 19, 2005 at 09:49 AM

• Preferences backed by a database

  Posted by: rbair on August 19, 2005 at 10:46 AM

• We use both Properties (for keeping track of save file information) and Preferences (for application-wide settings).

  Posted by: davidbrowne on August 19, 2005 at 11:11 AM

• HSQLDB is an interesting option; running it as an embedded relational DB.

  Posted by: rjlorimer on August 19, 2005 at 12:37 PM

• Hi Chet,

  I've used Preferences, Properties and XStream for application data.

  I always use Preferences to store things like window positions, window sizes, etc...

  I use Properties or XStream to store settings that could be loaded by the user to change the application behaviour. For example, project settings, etc... If it's real simple then I stick to Properties, but once it moves beyond that, I always use XStream.

  Posted by: jon_lipsky on August 19, 2005 at 01:55 PM

• Go Binary

  Posted by: chakrayadavalli on August 19, 2005 at 02:27 PM

• Preferences for standalone applications. For applets, I'll be pushing the data back to the server. Would like to set a cookie. Hint Hint.
  

  Posted by: neilweber on August 19, 2005 at 03:21 PM

• 
  I do http://www.hermesjms.com

  I use JAXB, its nice to work with a properly typed model (and I use Jakarta BeanUtils/PropertyUtils when it needs to be dynamic such as creating property sheets etc).

  I don't really care for JAXB (1.X, not tried the new stuff yet) as it has no query capability and I have to worry about the non transactional nature of the filesystem. I want to hide the fact that its XML as one day I want to change to something with an embedded database (I already use Derby for storing messages so maybe the same schema with Hibernate or similar?)

  I'm still looking for a good way to capture non-trivial configuration, both in Hermes and my day job. Huge XML documents are painful - even when there is a schema (when often there is none...)

  Luckily however I use JIDE which means frame layout is not my problem :-)

  Regards....

  Colin.

  Posted by: colincrist on August 19, 2005 at 03:42 PM

• Because I am using my application framework which is built around my plugin engine (www.platonos.org), I have a plugin that provides configuration management. It provides an extension point for plugins to extend to contribute new ways of storing things like preferences (user and global), as well as an extension point for plugins to extend and provide an implementation of the configuration interface so the plugin can store stuff. The nice thing is, I can add Preferences, Properties, XML, database or other ideas and the plugins that use the configuration plugin for storing stuff don't care how its stored. It's application defined as to what storage mechanism is used and is completely dynamic.
  

  Posted by: buckman1 on August 19, 2005 at 10:09 PM

• I used the JDBC api to get the config from a database.

  Posted by: urddd on August 19, 2005 at 10:23 PM

• Hi Chet,
  we are happy with the preferences API in JDK 1.4, although we are still using ascii files (both, property and xml files) in most of our projects.
  In many projects we have to store user config in a central database because this allows us to manipulate the configuration by the central support team (in order to help users when they struggle with the config ;-) )
  One downside of the preferences API on Windows is, that preferences can't be stored into files since all data goes to the registry. Files are better than the registry because you can make backup copies easily or send them by mail as part of an automated crash report. It would be nice if you could ship future versions of the Windows JDK with an additional PreferencesFactory which works similar to the Unix PreferencesFactory which stores preferences in the user's home directory (or in the global /etc).
  BTW: The preferences API looks very extensible to me now. Are there implementations of PreferencesFactory and AbstractPreferences for home-directory or database storing available on the net, preferably as free opensource libs?
  

  Posted by: prange on August 20, 2005 at 03:05 AM

• All my apps use the preferences API to store those settings; the UICompiler Mainwindow remembers its window size between sessions, for example. All done using the preferences API.
  You should note that on Linux this by default is an xml based storage :-)

  Posted by: zander on August 20, 2005 at 12:55 PM

• What we're looking for is:
  1. Array Based Properties file, like borland.jbcl.util.ArrayResourceBundle. it's not only startup realated, but is very important for performance. see "Other Resource Bundles" part in: http://java.sun.com/developer/technicalArticles/Intl/ResourceBundles/

  2. a way to have several attributes (like "type") for each resource in the resource bundle. So, having the "label", "tooltip", "mnemonic key" etc as a single resource for a button, instead of several resources cuts the search population by 70-90 percent.

  Not having those features pushes us to impelement our own resource bundles.
  cheers

  Posted by: gideosh7 on August 20, 2005 at 11:53 PM

• What we're looking for is:
  1. Array Based Properties file, like borland.jbcl.util.ArrayResourceBundle. it's not only startup realated, but is very important for performance. see "Other Resource Bundles" part in: http://java.sun.com/developer/technicalArticles/Intl/ResourceBundles/

  2. a way to have several attributes (like "type") for each resource in the resource bundle. So, having the "label", "tooltip", "mnemonic key" etc as a single resource for a button, instead of several resources cuts the search population by 70-90 percent.

  Not having those features pushes us to impelement our own resource bundles.
  cheers

  Posted by: gideosh7 on August 20, 2005 at 11:54 PM

• Yeah, there are lots of different ways to persist things. But here's an issue I haven't heard discussed: How do you securely persist passwords?

  Yes, of course they can be encrypted. But if you have an unattended batch application that needs a password to connect to a database, how does it decrypt the password? By a key? Then you the have the same problem: How do you secure the key?

  You certainly wouldn't want to hard code a password or key into an application. Not only might they change in the future, but if the code goes into a repository, the password/key is then available for other coders or hackers to read.

  Posted by: dadams07 on August 21, 2005 at 12:55 PM

• Hey Chet

  Everything we build is usually database orientated so all user based prefs are stored on the database and pulled into the app with JDBC api and all appication specific prefs are stored using the Preferences API.

  Posted by: ruxton on August 21, 2005 at 07:46 PM

• We also often use the Preferences API, with also sometimes property files or xml files. For XML we usually use SAX which is much faster than DOM.

  In a previous job we were also using the preferences API, for which we wrote several implementations, including an XML one since the one provided for linux is only available on Linux... and it's a shame, having a consistent implementation of Preferences betwen platform would be really nice.

  BTW I think that changing default implementation of Preferences should be easier: as of jdk 1.4 it was only possible to change it before the first time the API was actually used...

  Posted by: xhanin on August 22, 2005 at 02:57 AM

• Rolled my own (poorly) encrypted flat file.

  Posted by: heaththegreat on August 22, 2005 at 05:52 AM

• We converted from properties files to Apache Commons Configuration, because we have a lot of different properties files and wanted flexibility to possibly change a DB based configuration without changing the API.

  Posted by: denisk on August 22, 2005 at 06:00 AM

• We are using Apache Commons Configuration here. It is easy to use and very, very flexible.

  Posted by: dediana on August 22, 2005 at 06:19 AM

• Hey Chet,
  we are using the JavaBean persistence mechanmism (XMLEncoder & XMLDecoder), which is sometimes very slow and crappy.
  David

  Posted by: dlinsin on August 22, 2005 at 07:04 AM

• Hi,

  good topic! I prefer to use preferences to store the path of the properties file with all application settings. I also use command line options with the same name to allow the user to overwrite any setting individually.

  The real challenge to this was an API that allows me to use all three sources of application settings (preferences, properties and the command line) consistently. I developed my own API for this task, which is now available at http://truesetting.dev.java.net. Loading and parsing application settings from all three sources is now a matter of just a few lines of source code, basically a definition of one object for every setting.

  If you like to see this in action, please download the trial version of my application TrueMirror with JWS. The online manual explains the settings and how they are loaded from the properties and the command line.

  With best regards,
  Christian Schlichtherle

  Posted by: christian_schlichtherle on August 22, 2005 at 08:00 AM

• We use properties files here chet. I would like to look into the preferences api in the future, but for now this is what we use.

  Posted by: haskovec on August 22, 2005 at 08:26 AM

• Interesting question. Given that my work tends toward more standalone UI's, I usually pass on using a database to store this level of application/user settings (besides, even if you do use a database, you still have to worry about where to get the connection info from and I hate having to bother the user with mundane stuff like that). I also pass on using the preferences API. During development and testing, the information isn't accessible enough for me (not bad under Linux, but a pain under Windows) and during production support, it's almost impossible to get relevant information from the user unless I actually build the collector for that info into the UI.
  For the longest time I used Properties files, but I'm finding that the typical UI these days requires the persistence of data more complex than properties are suited for. When I started spending a lot of time working out nifty encoding schemes for the values of the properties, ... well, you get the idea. I have occasionally used object streams, but, again for debugging and support purposes, I prefer a readable text file to binary stuff I need special tools to examine. This also applies to embedded databases, though, at least for now, their additional size requirements bother me even more.
  This has let me to my current preference (no pun intended) of using XML. A couple of things to note. I never use schemas/DTD's; it's just not worth the hassle and I can validate things easy enough given the size/complexit of XML documents involved. I have distilled what I need down to two classes (XmlElement and XmlUtils). The main thing that has bothered me about XML support is two-fold. Since all the libraries (from the ones in Java itself to those provided by folks like apache) are, by definition generic, they come with far more weight than I want (so thrid party libraries are out; I'm not going to burden my users with 150K jar file just to give them persistent settings). The flip side is that the support provided in Java comes in two extremes. I have SAX, which is (relatively) light, fast and memory efficient, but takes a fair amount of coding to use and I have DOM, which is a HUGE memory hog. Now, I have never, epecially in this context, needed to recreate the XML document *exactly* as it was defined in the file. So. XmlUtils has the stuff I need to read/write XML documents (to/fromString() and to/fromFile() methods are used a lot) and XmlElement represents my node structure.
  My main suggestion would be to provide one or two "bridge" classes (like XmlUtils and XmlElement) that would provide *slightly* higher level support than SAX, but require neither validation nor exact representation (as DOM does).
  Sorry to be so long-winded, but I hope you find this helpful.

  Posted by: jskress on August 22, 2005 at 09:03 AM

• Properties file.

  Posted by: gdusbabek on August 22, 2005 at 11:13 AM

• Here too, properties file.

  Posted by: leocervo on August 22, 2005 at 12:11 PM

• I use the Preference API, with the preferences backed in a XML file (custom Preference factory)

  Posted by: magnum on August 22, 2005 at 02:42 PM

• Preferences backed by XML if platform == Windows, use JRE's default factory otherwise. Also have an option to explicitly export/import the prefs using the API's own facilities for this.

  On the toppic of persiting passwords --> DON'T. You cannot. It's an oximoron, or a hall-of-mirrors problem with keys protected by keys which ends in a hardware protected key...

  If you need unattended access to password-protected resources, you are far better off caching what you can to a non-protected location. Your protected resources should not need to be accessed by an automation. If they do, then can you still find some way to separate them, or limit privilege to the automation as much as possible at least, so that if it's password is compromised your risk is small? That is what is done in the financial industry. Either that, or you weight the financial risk of a compromise against the cost of paying a monkey to attend the batch (which is what I think the military do).

  Posted by: sinewalker on August 22, 2005 at 06:37 PM

• We roll our own files, if only for legacy reasons. We are looking into the Preferences API and possibly XML.

  Posted by: maiku on August 22, 2005 at 10:33 PM

• I am definitely switching to XML and phasing out Preferences and Properties. The writeObject() and readObject() facilities of the Serializable interface allow me to formulate the persistent data as fields of a persistent object (or a hierarchy of). It's neat, very light on code, easily documented, and very programmer friendly. It survives well my frequent modifications to the project.

  I am also using a legacy format that's rarely heard about: Fortran Namelists ! That's because a lot of my work goes into building neat gui's for engineering calculations. Previously, Namelist files were my main persistent medium. But that's over now thanks to the ease and scope of XML Serialization. Namelists are now used only for transmitting data to the Fortran software. The gui's connect the user to a lot more resources than just the input parameters for a given calculation. Now, thank to XML, this information is easily managed, free of the Fortran constraints, and is expanding in scope.

  Btw, I'm curious about if / how other people drive legacy scientific software with Java gui's...

  Posted by: zed_b on August 24, 2005 at 09:07 PM

• I am definitely switching to XML and phasing out Preferences and Properties. The writeObject() and readObject() facilities of the Serializable interface allow me to formulate the persistent data as fields of a persistent object (or a hierarchy of). It's neat, very light on code, easily documented, and very programmer friendly. It survives well my frequent modifications to the project.

  I am also using a legacy format that's rarely heard about: Fortran Namelists ! That's because a lot of my work goes into building neat gui's for engineering calculations. Previously, Namelist files were my main persistent medium. But that's over now thanks to the ease and scope of XML Serialization. Namelists are now used only for transmitting data to the Fortran software. The gui's connect the user to a lot more resources than just the input parameters for a given calculation. Now, thank to XML, this information is easily managed, free of the Fortran constraints, and is expanding in scope.

  Btw, I'm curious about if / how other people drive legacy scientific software with Java gui's...

  Posted by: zed_b on August 24, 2005 at 09:09 PM

• I always show up late for the blogs!

  Hi.

  I would like to strongly -- wave-my-arms-around style -- chime in with what I see as the need for some type of support as posted above by prange: Where we could specify in the Prefs API, our preference for using either files (in user home) or in the registry/OS-chosen location -- maybe with a fallback.

  The main reason is that the user can back this stuff up without aching about Windows-specific backup tools that will back up the registry. And not to mention the myriad ways the %$ registry gets corrupted seemingly every day in a new way.

  The users would benefit from this one! They would get to choose. And we could use the unified Preferences API for storage here, or storage there.

  In any case: Thanks for asking!

  Later,
  Steev.

  Posted by: steevcoco on August 27, 2005 at 02:20 PM

• Hi,
  for saving Client Application Data and Preferences I'm using one or more XML Files (using for example XStream) but compressed, like OpenOffice does with his Documents.

  I'm using also Prevayler (but mainly on Server side), a very interesting persistence library that uses this (and others too) approach with a lot af data, you can find here: www.prevayler.org

  Bye,
  Sandro
  

  Posted by: smartini on September 01, 2005 at 02:46 AM

• Serialized HashMap, altough I have written an improved seralizer that store primitives/Strings in a very compact way... all binary of course...

  Posted by: terifan on September 09, 2005 at 01:17 AM

This work is licensed under a Creative Commons License.