The Source for Java Technology Collaboration
User: Password:



Stanley Ho

Stanley Ho's Blog

More JSR-277 Early Draft Q & A

Posted by stanleyh on November 03, 2006 at 10:51 AM | Comments (8)

Since the JSR-277 early draft became available, many comments have been raised by some reviewers. Some of the comments are valid concerns and good suggestions, and these are good inputs that will be discussed in the EG (Expert Group). On the other hand, some other are misconception and misinformation, and that's what I would like to address in this blog.

Before we begin, let me remind everyone that JSR-277 is only in the early draft stage, and it is by no means complete. The design in the early draft is deliberately kept simple, so we can make the spec available to the Java community sooner for feedbacks. Thus, it should be no surprise if you find issues or holes in the early draft (otherwise, it's not really an "early" draft anymore, right?). In any case, if you have any feedback on the early draft, please send them to jsr-277-comments@jcp.org.

Now, are you ready? Here we go:

Q: What are the advantages of JSR-277 compared to other existing systems?

Many existing systems (e.g. .NET, OSGi, NetBeans, Maven, Ivy, etc.) offer reasonable solutions in their problem space, but they are not trying to solve the same problem as JSR-277. Because JSR-277 will become part of the core Java platform, it has several distinct advantages that allows us to solve problems in a different (and we think more elegant) ways than existing systems, for example,

* It can, in theory at least, define a better solution since it has the ability to mandate deep changes to the underlying architecture of the SE platform, and in particular class loading, which is something that no previous module system could do.
* It can integrate with JSR-294, which will move module definitions from crufty XML files into the Java language itself. Kind of like replacing Java EE 1.4 deployment descriptors with in-code annotations in Java EE 5.
* It is extensible. That should make it possible for other systems to interact with the Java Module System and even make it possible for them to offer new features made possible by the integration of JSR-277 into the Java platform. We are still working out many of the details and would appreciate feedback on this topic.

To learn from the experience of existing systems, JSR-277 has a large (20 members) and very diverse EG to accommodate a variety of viewpoints (e.g. a few members are .NET experts, two are OSGi experts who co-authored the R4 modularity spec, one is Maven's creator and one is Maven expert, one is Ivy's creator, etc.).

Q: Does the module system do built-in type consistency checking?

Yes. After each module instance is connected to all of its imports, shallow validation is performed on the module instance. One of the checks shallow validation will go through is to ensure that there is no overlap in the exported types from imported modules. Let's look at an example,

A import B v1.0, C v1.0
B import D v1.1
C import D v1.0

Suppose type T is in D. If type T is neither re-exported from B or C, then T is not visible to A at all and there is no type inconsistency. If type T is re-exported only from B or only from C, there is still no type inconsistency. Only when type T is re-exported from both B and C, there is a type consistency issue, but it will be caught by shallow validation automatically and A will not be fully initialized.

It is true that if B returns an object of type T to A via a "cast" to a superclass such as java.lang.Object, and B passes that object to C with tries to cast it to T, then that will fail. However, this is a rare scenario in real applications and we expect that the deep validation would catch this kind of inconsistency.

It is still fair to say that type consistency checking is not explicitly spell out in the early draft, and it probably requires lots of refinements in the next few months. If you have any good suggestion to contribute in this area, I'm sure the EG would appreciate your inputs.

Q: Can modules be loaded or unloaded at runtime? Does it require a JVM restart?

Yes, modules can be loaded or unloaded at runtime.

New modules can be installed into a repository at runtime for the module system to use and load them. Existing modules can also be uninstalled from the repository at runtime so they will not be used when other modules request them subsequently.

When a new repository is created at runtime, modules from the new repository will become available to the module system dynamically. When a repository is shutdown, all modules will be torn down so they can be unloaded eventually. (See Section C.10 in the early draft - how to tear down a module is a future work item for the EG). In fact, we expect module-based Java applets and JNLP applications will be supported by using the URLRepository class, so web-based modules can be loaded and unloaded from a codebase dynamically in a JVM.

In any case, JVM restart is NOT required to load or unload modules.

Q: Mapping a module file (defined in JSR-294) to the module metadata (in JSR-277) seems like a huge constraint. Does it mean that the deployment module is rigidly bound to the development module?

Actually, I consider this degree of integration between JSR-277 and JSR-294 a big advantage. This allow us to move module definitions from crufty XML files into the Java language itself. This is kind of like replacing Java EE 1.4 deployment descriptors with in-code annotations in Java EE 5. Also, by specifying the module information directly in the source file, the compiler can validate the module information to ensure its correctness (How many hours did you spend on debugging manually written metadata?); it'll also be possible for developers to compile a module and generate a JAM file for deployment in a single step - without going through a separate step for writing separate metadata and packaging like what we have been doing for years.

Q: Why didn't the spec use a XML representation format for module metadata?

The format for the module metadata will be defined in JSR-294, and the actual format has not yet been decided by the JSR-294 EG. What is shown in the JSR-277 early draft is just a logical representation of the information in the module metadata file and should not be mistaken for the physical file format. Make no mistake, XML is a reasonable representation format for typical metadata. On the other hand, because the module metadata will be used directly by the JVM to enforce module membership and module-level access control, I won't be surprised if the JSR-294 EG decides to use a more compact representation format.

Q: Does JSR-277 have split package problem?

Split package refers to the issue that classes from the same package can come from different modules through the import. We currently do not plan to support split packages in JSR-277. It will be spelled out more explicitly in the JSR-277 specification in the future.

Q: Isn't specifying the import policy declaratively in the metadata a better choice than expressing the import policy in Java code?

Not necessarily. Designing declarative metadata is typically based on a set of requirements. However, as soon as there are new requirements arise that the existing design can no longer satisfy these requirements, the declarative metadata will need to be extended. Imagine if you want to use a module with metadata v2 in a module system that only recognizes metadata v1, you'll likely run into some problems. Import policy has no such problem because developers can express any logic they want in the code as long as the programming language and the class libraries support it, and the import policy can be executed and supported by all versions of the module system.

It's a valid concern that some import policies may not be written properly thus may cause side effect to the module system. However, it is not much different from encountering problems while executing code in the static initializer or instance initializer today. Even if the import policy results in exception during execution, this is really not an issue as long as the JVM can recover gracefully from it.

It's also a valid concern that expressing the import policy in Java code may be difficult for tools and management systems to analyze module dependencies, and this will be addressed in the next question.

Q: Without declarative metadata, it seems impossible for tools and management systems to analyze module dependencies.

It is not uncommon for the module system to have many repositories at runtime. Contents in the repositories are opaque, and they could be stored in different media (for instance, file system, database, HTTP server); the Repository APIs would effectively hides that from the application/tool. In addition, new repositories could be created and existing repositories could be removed at runtime. Also, it is possible that some implementations of the module system could provide mechanism to override/block/redirect module dependencies based on some deployment policies. Given there are so many potential variations in deployment, if tools and management systems attempts to analyze the module dependencies by accessing the physical metadata from the modules in the repositories, the task would be very complicated. Perhaps asking a different question would help:

Does it really make sense for tools and management systems to parse all the module metadata from the repositories and analyze module dependencies on their own, given all the capabilities in accessing the modules from the repositories and in supporting any potential override mechanism are already built into the module system?

Obviously, the answer is no. In fact, if the module system can provide APIs to expose dependency information it already has for a specific module, tools and management systems can simply make use of the dependency information provided by the module system. Hence, it is not necessary for module metadata to be declarative in order for tools and management systems to analyze module dependencies. As mentioned in Section C.5 in the early draft, missing dependency prediction is one of the future work items for the EG.

Q: Does the module system support module lifecycle?

The early draft specification does not spell out any lifecycle support in details, but it is on the EG's radar. As mentioned in Section C.6 in the early draft, more explicit lifecycle is one of the future work items. If you have any requirements on this feature, I'm sure the EG would like to hear them.

Q: Does the module system support optional import dependency?

Optional import dependency is currently not supported in the early draft. However, the EG may consider such feature if there are reasonable use cases and sufficient demands. If you have any requirements, I'm sure the EG would appreciate your feedbacks.

Q: Is the built-in repository model too simplistic?

Yes, but for good reasons. One of the strengths of the Java Module System is its extensibility. It makes it possible for 3rd parties to plug in custom repository implementations to address complex or unforeseeable cases when these cases arise. Hence, we can keep the built-in repository model very simple, and it doesn't have to deal with every possible use cases.

Q: The early draft does not specify how the module-aware compiler should compile the module sources.

The language aspect and compilation aspect of a module will be defined in JSR-294, instead of JSR-277. Having said that, we will likely provide more explicit instructions in the future for module-aware compilers to generate versioning constraints properly when compiling JSR-294 module in the context of JSR-277.


Bookmark blog post: del.icio.us del.icio.us Digg Digg DZone DZone Furl Furl Reddit Reddit
Comments
Comments are listed in date ascending order (oldest first) | Post Comment

  • I'm glad people are taking this issue seriously. The classpath can be such a nightmare sometimes!
    Is this finally going to fix the infamous:
    Exception in thread "main" java.lang.NoClassDefFoundError:

    Posted by: firefight on November 06, 2006 at 06:08 AM

  • Stanley, with respect, I think you've dodged the first question rather than given a proper answer. What set of problems is 277 solving that cannot be solved with OSGi R4? You only list a few things that 277 will have in its arsenal to solve whatever problems it's supposed to be solving, but you haven't made explicit what the problems actually are. Taking those points one by one:


    What class loading scenarios are possible with 277 that are not possible with the OSGi class loaders? Beyond cleaning up some of the bugs in Sun's VM implementation (eg 4670071) there appears to be little that can be added... OSGi works, after all.

    Even assuming that moving all this data into Annotations is a good idea (not necessarily the case), why can this not be done with build tools? Module definitions in the form of XML or OSGi Manifests could easily be generated using apt.

    OSGi is extensible, Maven is extensible, Ivy is extensible. What specific extension scenarios are you looking at that cannot be achieved without deep JVM support?


    Regarding the lack of declarative metadata, I still think this is a mistake, and I'm sure some of your colleagues on the EG agree, whether they have said so in public or not.

    Fine, the repository may be opaque, and having a Repository API is a good idea to avoid many tools having to implement parsers, dependency calculators etc. But why does the module metadata have to be opaque with respect to the repository itself, ie why the need for executable code within each module to perform dependency resolution? The modules should provide transparent, declarative metadata and then the Repository API should implement the parsers, dependency calculators etc for use by the tools.

    Also the point made by Peter Kriens about the environment in which the resolution code runs was a good one. By definition, that code must run before the module is resolved. So how does the tool (or Repo API) supply it with a class loader? What external dependencies does the resolution class have, and how on earth can they be supplied?

    Programmatic code is unstable and potentially non-repeatable. A module author could easily write a resolver that only resolves the module on Tuesdays... so a tool tells you the module graph is complete, but when you run it nothing happens because it's now 00:02 on Wednesday. Alternatively the resolver code could NPE or enter an infinite loop. Would this lock up the main application thread? Sounds like a potential DOS attack: just drop a module into a public repo with a "while(true){}" in its resolver code.

    Okay okay, this may be frivolous... but it illustrates the problem with programmatic resolution: it's too flexible. Declarative is simple and repeatable: "if module X version 1.2 and module Y version 3.1 are both available, and this is a Linux box, then I am available". What more do you need?

    NB: For those people reading this blog and thinking "modular Java sounds like a great idea, why knock it?"... actually you're right! But you don't have to wait for Java 7 to try it out, give OSGi a try, it works with the JVM you're running now (and yes, it will still work when JSR 277 is eventually available!)

    Posted by: njbartlett on November 06, 2006 at 08:47 AM

  • I agree with the expert group, and think njbartlett is crazy.

    OSGi is fantastic, but XML blows. I don't care about the bazillion virtues of declaring things in XML, I think the features would best exist as the author described them. I want my compiler to report problems. I don't want to dig through a mangled OSGi XML file where the slightest mistype in the file will break the schema and cause it to fail. I don't want a file that was corrupted by my IDE after I tried to make some by-hand changes.

    You say that "Programmatic code is unstable and potentially non-repeatable". That is the must nonsensical thing I've hear. Actually, it's quite the reverse. Code is perfectly repeatable and easy to debug and manage. XML is a big goopie mess of ASCII text that's slow, verbose, and more error-prone compared to code.

    I want a little red squiggly line under my incorrect code as I type. I don't want to deal with an onslaught of runtime errors with a stupid XML file not parsing correctly.

    XML is evil. JSR 277 is doing the right thing. Accept it.

    -Bryan

    Posted by: prime21 on November 06, 2006 at 03:23 PM

  • Bryan,

    I agree with you that XML "blows". So do the OSGi people. That's why there is no XML in the OSGi module metadata.

    However even if there were, XML is at least possible to introspect statically, whereas programs are not. Of course other formats (eg the format actually used by OSGi) are even easier to handle.

    I encourage you to read up on OSGi. If you're prejudiced against it because you don't like XML then I believe you will be pleasantly surprised.

    (BTW that red squiggly line when you type broken Java code requires a lot of very clever on-the-fly compilation support in your IDE, and a fair amount of work on your part to get the dependencies configured right. You can get tools that do exactly the same for XML and other structured metadata formats, and they are far simpler and less error-prone. Plugins exist for this in all the major IDEs).

    Regards, Neil

    Posted by: njbartlett on November 06, 2006 at 05:02 PM

  • I agree with you that XML "blows". So do the OSGi people.

    Err, I don't mean that the OSGi people "blow"! Nor do they necessarily think XML "blows". It just isn't used for the core module metadata, mainly because of the speed and verbosity issues.

    Posted by: njbartlett on November 06, 2006 at 05:12 PM

  • I agree with you that XML "blows". So do the OSGi people.

    Err, I don't mean that the OSGi people "blow"! Nor do they necessarily think XML "blows". It just isn't used for the core module metadata, mainly because of the speed and verbosity issues.

    Posted by: njbartlett on November 06, 2006 at 05:16 PM

  • thanks, this helps clarify things

    Posted by: asjf on November 07, 2006 at 01:59 AM

  • njbartlett: There are many existing and competing systems today. The whole point of JSR-277 is to learn from the experience of these systems and synthesize the best attributes of these technologies to produce a module system for the next generation Java SE platform that would benefit everyone in the Java community, with first-class modularity, packaging, deployment, built-in language (with JSR-294), classes libraries and JVM support.

    I have no doubt that many existing systems offer reasonable solutions in their problem space, but it doesn't mean they are trying to solve the same set of problems as JSR-277 along with JSR-294 (e.g. information hiding, classpath hell, jar hell, extension hell, etc. - these are not comprehensively addressed by any existing system; in fact, they all suffer from these problems in some degrees).

    Without addressing the classloading issues, many existing systems can run into deadlock in multi-threading environment. Fixing 4670071 is not simple, because the fix we have seen so far may introduce backward incompatibility and break existing applications. What we want is a more general solution (not a bug fix) in Java SE 7, so existing applications will continue to run with existing classloading behavior, while some systems that require parallel classloading will be able to take advantages of the new capability. This obviously requires deep changes in the Java Runtime.

    For the metadata, the point is that we would like to make it easy for end-to-end development/deployment for module developers. Putting the module information in the source obviously makes it simple for developers since everything will be in one place, and the compiler/tool can validate the info and generate/update the metadata transparently. Of course, this can be done in many other ways, but that's not the point.

    The architecture of the Java Module System is extensible that allows 3rd parties to plug in alternate implementations for the repository mechanism and for the actual module system implementation. Thus, many existing systems (e.g. OSGi, NetBeans, Maven, Ivy, etc.) can extend the Java Module System and add values. Also, the investment that developers/companies devoted to these existing systems can be preserved.

    For the custom import policy, the problem is that some people may not be happy with strictly constant and fixed imports. The choice was between defining a custom declarative language that captures each use case that somebody insists is important or to tell people to write Java code for those unusual cases. That said, we expect to cover majority of scenarios without a custom import policy, and you can statically analyze those modules.

    Anyway, we are still working out many of the details in the specification and would appreciate feedbacks from the Java community.

    Posted by: stanleyh on November 17, 2006 at 12:45 PM



Only logged in users may post comments. Login Here.


Powered by
Movable Type 3.01D
 Feed java.net RSS Feeds