The Source for Java Technology Collaboration
User: Password:



Stanley Ho

Stanley Ho's Blog

JSR-277 Early Draft Specification

Posted by stanleyh on October 27, 2006 at 06:00 AM | Comments (18)

The JSR-277 early draft specification has recently been published - this is what I've been spending significant amount of time during the past year, and I'm very glad that it is finally available to the Java community for review.

What is JSR-277?

JSR-277 (Java Module System) is an architecture for the development and deployment of module-based applications and libraries. Applications and libraries written as Java modules are easier to develop and deploy, with first-class modularity, packaging, and deployment support in the Java SE platform.

This architecture defines the following components:

  • A distribution format (i.e., a Java module) and its metadata as a unit of delivery for packaging collections of Java classes and related resources. The metadata contains information about the module, classes and resources within the module, and its dependencies upon other modules. The metadata also includes list of exports to restrict classes and resources from being exposed outside the module unintentionally.

  • A versioning scheme that defines how a module declares its own version as well its versioned dependencies upon other modules.

  • A repository for storing, discovering, and retrieving modules with versioning and isolation support.

  • Runtime support in the application launcher and the class loaders for discovering, loading, and integrity checking of modules.

JSR-277 is currently targeted for Java SE 7.

What were the motivations behind this JSR?

Java Archives (JARs) are widely used as both the distribution format and the execution format for Java applications. The JAR file format dates back to the mid-1990s, and it has not scaled particularly well in either of these roles. JAR files are hard to distribute, hard to version, and hard to reference in general.

Distributing a simple Java application is considered to be a complicated task by many developers because it often involves creating a native installer to package multiple JAR files into a distribution unit, and it sometimes involves converting the application into a Java applet or JNLP (Java Network Launching Protocol) application for web-based deployment.

The versioning support in the JAR file format is for expressing dependencies on Java extensions (a.k.a optional packages), but not the version of the JAR file itself. There is also no reliable mechanism for expressing, resolving, and enforcing the dependency of one JAR file upon another. Referencing a JAR file, moreover, involves specifying it in the classpath. Since the path of a JAR file may change during deployment, developers are forced to fix up all the references to the deployed JAR files as part of the deployment process.

Developers also find it quite difficult to deploy installed Java extensions because they can easily run into issues like versioning conflict and namespace collision. It is also basically impossible to arrange for multiple versions of installed extension to be shared across multiple JRE installations.

These issues have been collectively known as the Classpath Hell, the JAR Hell, and the Extension Hell for years. The overall goal of the JSR-277 is to define first-class modularity, packaging, and deployment support in the SE platform that are capable to address these issues once and for all.

What makes JSR-277 so different compared to existing systems?

This is definitely one of the most frequently asked questions. The complete answer would never fit into a blog, but there are several points I want to highlight:

  • JSR-277 is generally less complex.

    We deliberately made the design simpler by following this principle - make simple things simple, and make complex things possible. Because JSR-277 is targeted for Java SE 7, we have flexibility to update the JVM and classes libraries in the SE platform to better support JSR-277, thus reducing the overall design complexities. Also, the JSR-277 architecture is very extensible - 3rd parties may plug custom implementations to address complex or unforeseeable cases when these cases arise, so the core architecture can be kept simple.

    Note: It is also fair to say that the module system described in the early draft is relatively simple because the early draft is by no means complete.

  • JSR-277 has powerful repository.

    Repository is a mechanism for discovering, storing, and retrieving modules, and multiple versions of modules can be deployed side-by-side in a repository. Typically, the module system has one or more repositories at runtime, and the repository architecture is very extensible. Let's look at some implications:

    • Multiple repositories could be used to achieve different degree of module sharing in the corporate environment - an per-user repository may be provided to share modules across all JREs but specifically for the current user, and a global repository may be provided by the system administrators to share modules across all JREs for all users within the corporate environment.
    • URL repository could be used to expose modules from a specific codebase on the server. Module could be downloaded and deployed on demand when the module is needed at runtime.
    • Custom repository implementations could be easily plugged into the module system. The storage model in the repository is abstract, so it is possible to implement the repository using different media, for instance, file system, database, etc. It is also possible to implement the repository using existing repository systems as back-ends, e.g. Ivy and Maven.
    • Module must be deployed into a repository before the module system can discover and make use of it. This deployment step offers repository implementation an opportunity to transform a module into a more efficient format for run time access. For instance,
      • Index can be built for all classes and resources in a module to optimize classloading at runtime.
      • Native code can be generated for the frequently used classes in a module to optimize runtime performance.
      • Byte code in a module can be verified at deployment time, thus eliminating the impact of byte code verification in a module at runtime.
      • Digital signature in a module can be verified and the certificates in the digital signature can be extracted at deployment time, thus eliminating the impact of digital signature verification in a module at runtime.
  • JSR-277 has powerful reflective APIs.

    Key abstractions in JSR-277, e.g. module definition, module instance, repository, etc., are all reified through the reflective APIs. This enables the module system to expose functionalities to everyone in a very consistent manner. Suppose the Java launcher needs to access the classloader object of a specific version of a module from the repository and an application needs to access the same thing - the Java launcher and the application simply invoke the reflective APIs to get what they want from the module system.

  • VM support for module-level access control (through JSR-277 integration with JSR-294)

    Each module defines exports that defines what classes and resources that are visible to other modules. At runtime, module-level access control is enforced at the VM level, thus keeping the Java platform secure by disallowing other modules to access module-private/non-exported classes in another module.

  • Development and deployment integration (through JSR-277 integration with JSR-294)

    In typical module development, developers would start from writing a development module (defined in JSR-294), then compile and package the artifacts into a deployment module (defined in JSR-277). Basically, the development module is the source form of the metadata in the deployment module, and there are several implications:

    • The metadata in the deployment module is generated by compiling the development module. If there are errors in the development module, they would be caught by the compiler. Therefore, it significantly reduces the likelihood of malformed metadata in the deployment module.
    • Whenever there are code changes in Java classes that belongs to the development module, recompiling these classes would force the recompilation of the corresponding development module. In other words, the metadata of the deployment module is always kept up-to-dated automatically whenever the code that belongs to the development module is recompiled, thus freeing the developers from doing manual bookkeeping on the metadata.
    • This also simplifies development and the build process (also IDEs) because everything is in the Java source files.

    In addition, exports in the deployment module are declared in the development module. If a class in a module attempts to access module-private/non-exported classes in another module, the error would be caught early on by the compiler during development rather than causing failure at runtime.

  • Extensibility

    A trademark of many Java platform APIs is extensibility and allows 3rd parties to deliver innovations that plug into the JRE. For example, ClassLoaders, service providers, etc. JSR-277 is not only a module system by itself, but it is also an architecture that is extensible by other module systems (e.g. OSGi, NetBeans, etc.). Although the details still need to be worked out, the goal is to allow some degrees of interoperability between JSR-277 and other module systems. By providing custom module and repository implementations, other module systems may expose their modules through the repository for other JSR-277 modules to import. Similarly, through the powerful JSR-277 reflective APIs, other module systems may also allow their modules to import JSR-277 modules. A more extreme case would be for two or more other module systems to expose their modules as JSR-277 modules through the repository, and the framework in JSR-277 would enable modules from these module systems to interoperate with each others.

  • JSR-277 has integrated JSR-200 (Pack200) support for its distribution format

    JSR-277 defines a distribution format for module called JAM (JAva Module), and it is based on the JAR file format. JSR-200 (Pack200) defines a hyper-compression scheme for the JAR file format. Hence, developers may use Pack200 to further compress modules for distribution. The module system supports packed module archives directly; a packed module archive is automatically unpacked if it is provided to the module system. Thus, JSR-277 module is typically packed for distribution, and its small physical size makes it very convenience to distribute, especially for download.

What should you do?

I hope you now have more ideas of what JSR-277 is about, and I encourage you to learn more by reading the early draft specification. The EG definitely would like to gather constructive feedback on the directions and the open issues indicated in the early draft. If you have comments, please send them to jsr-277-comments@jcp.org.



Comments
Comments are listed in date ascending order (oldest first) | Post Comment

  • There has been some disagreements regarding the specification. Especially, this review "http://www.osgi.org/blog/2006/10/jsr-277-review.html" seems to be raising lots of questions. Is it possible for you to provide answers to the issues reaised in the above review in a point-by-point reply? This would atleast reassure us that this specification is not going to be another EJB debacle. Rejection of this spec would really put componentization of Java one or two years back.

    Posted by: vhi on October 27, 2006 at 07:45 AM

  • In addition to above, please also look into "http://robilad.livejournal.com/1672.html". It voices exactly the concerns I have. Please engage the community and others who have sufficient experience in this area. This is a very important spec for Java, and is going to be the foundation for many more future technologies. Screwing this up would endanger Java's future itself.

    Posted by: vhi on October 27, 2006 at 07:56 AM

  • Please read this review. I totally agree with what is being said there.
    Take it into account, please.


    http://weblogs.java.net/blog/patrikbeno/archive/2006/10/jsr_277_review_1.html

    Posted by: imjames on October 27, 2006 at 09:32 AM

  • vhi and imjames: These are interesting reviews; I think some of the issues are already answered in this blog, and I am going to follow up other issues and provide answers for them.

    Posted by: stanleyh on October 27, 2006 at 12:08 PM

  • Some of the issues mentioned in robilad's blog are related to a public project (https://jsr-277-eg.dev.java.net/). Actually, this project was initially setup when the EG was formed but the EG ended up using a different mechanism for collaboration. That's why the site and the mailing list is so unused. To avoid further confusion, this project has been removed.

    Posted by: stanleyh on October 27, 2006 at 12:10 PM

  • As I sure you are aware, there is some serious negativity about this JSR so far, focussed around the areas of closed EG process and perceived NIH syndrome. This seems particularly pointed when I find a link today from Peter Kriens indicating that he was specifically rejected from the JSR process despite being what looks like the perfect EG member (http://www.aqute.biz/Blog/2005-08-11). Please open up the process and bring in new ideas.

    Posted by: scolebourne on October 27, 2006 at 03:56 PM

  • Hi Stanley: thanks for giving us your perspective on the JSR. You wrote in a comment: "the EG ended up using a different mechanism for collaboration"--what mechanism is this, exactly? Where can we read the discussions, thoughts, debates and open issues regarding the JSR? Can you commit to an open, at least read-only discussion process? Knowing what the expert group is thinking and discussion might help answer one question your blog does not answer--what exactly the expert group thinks of Maven, Ivy, OSGi and other current, working approaches to repository and module distribution. Right now, from what I can tell, this looks to be yet another specification dropped on the laps of Java developers, who will have to pucker up and live with it, without knowing how exactly it came to take it's funny shape. You seem like a bright, hard-working guy, only hope you can be more open in leading the "community" process (from which the community seems to be strangely absent). Best regards, Patrick

    Posted by: pdoubleya on October 28, 2006 at 10:36 AM

  • but the EG ended up using a different mechanism for collaboration

    So you decided very early to go into typical JCP stealth mode. Understandable, since being on the forefront and having inside knowledge and understanding of this changes to Java is a huge competitive advantage. And we don't want to get that advantage diluted by having open, public discussions and development, do we?

    The 'C' in JCP was always a joke, 277 is just continuing the tradition of excluding the community. I a few years down the road we have to swallow 277. And everyone who will complain then will be told "but you could have joined the JCP and engage in 277 years ago". Sure, if I would have no real work to do.

    Posted by: ewin on October 29, 2006 at 01:26 AM

  • I also agree with others regarding the 'openness' of this JSR. I think it is a very important spec and exposing it to the general community and getting early feedbacks would be very helpful. Since Java is going on the path of Open Source, why not make atleast communication open with this spec?

    Posted by: vhi on October 30, 2006 at 04:10 AM

  • I also support the suggestion to make the discussions about this JSR as open as possible. I believe that his JSR is one of the most important changes in the future of Java, and it is critical to get it right from the beginning.

    Posted by: larswestergren on October 30, 2006 at 04:33 AM

  • Just a thought on the VM changes - does that expose potential / existing language implementations upon JVM to this new module system as well?

    If so - and I'm just wondering about it here, no complaints ;-) - shouldn't we wave hands to those developers as well? I mean, JSR is for Java Language and its libraries, but does it dictate the whole of JVM as well?

    Posted by: alexlamsl on October 30, 2006 at 04:43 AM

  • ewin: "So you decided very early to go into typical JCP stealth mode. ... Sure, if I would have no real work to do. " Perhaps you have provided the typical reason why so many JSRs end up in stealth mode. The expert groups also have real work to do and working in public is often more time consuming. I'd like to see a better balance --- too many expert groups spend far too long in purdah.

    Posted by: mthornton on October 30, 2006 at 08:58 AM

  • Hey!! that's not right. Something fishy is going on here. At one point there was at least 11 posts on the subject, all bringing up good points, and asking the tough questions. Where did they all go? Let's hope no one is trying to pull a fast one on us, and try to dodge the issues. Please put these posts back, this jsr is already in deep trouble as it is, and however inadvertent this act may be, it will only compound the problem.

    Posted by: mikeazzi on October 30, 2006 at 11:31 AM

  • I'm unconvinced as to whether the critics have valid concerns or not. But if there is any chance of this JSR becoming the next JSR47 then a robust defence of your design is essential. While the majority of Java developers have had the luxury of completely ignoring JSR47, we won't be able to ignore JSR277. Getting this wrong would be catastrophic.

    If you're confident in your choices, then you should have no problem responding to the many well-phrased concerns posted in response to the early draft.

    Posted by: brendonm on October 30, 2006 at 01:03 PM

  • While the outcome of JSR47 (logging) was undesireable, it isn't clear to me whose fault (if any) that was. The most sigificant difference between java.util.logging and log4j seems to be the point at which localisation is applied. In one the choice of locale is made where the log record is generated, while in the other it can be delayed to the point where the log is viewed.

    Posted by: mthornton on October 31, 2006 at 02:11 AM

  • pdoubleya, vhi, larswestergren: We hear you. We will be investigating ways to make the EG discussions more transparent to the community at large.

    Posted by: stanleyh on November 02, 2006 at 10:16 PM

  • alexlamsl: The linguistic aspect will be defined by JSR-294 (Improved Modularity Support in the Java Programming Language) instead of JSR-277. On the other hand, it is possible that JSR-277 may mandate deep changes to the underlying architecture of the SE platform, e.g. class loading.

    Posted by: stanleyh on November 02, 2006 at 10:23 PM

  • mikeazzi: This is a known issue in the blogging tool - when a blog is updated more than once, a different page is created instead of simply replacing the old one. Try http://weblogs.java.net/blog/stanleyh/archive/2006/10/jsr277_early_dr_1.html

    Posted by: stanleyh on November 02, 2006 at 10:25 PM



Only logged in users may post comments. Login Here.


Powered by
Movable Type 3.01D
 Feed java.net RSS Feeds