Skip to main content

Versioning: Can Be Done Right?

Posted by patrikbeno on May 30, 2008 at 9:56 AM PDT

JSR #277: Stanley Ho explains versioning... Sounds like a bad idea to me...


I have commented his post. Anyway, I think the idea deserves its own blog post: I hope to see some discussion about it. And I really wonder what your opinions are.

Here it is:


Major? Minor? Micro? Update? 3 or 4 numbers?

IMHO, doesn't matter at all...

Stanley, what happened to or what is wrong with my proposal of versioning?

With proposed solution (version compatibility based on version history), module system would not have to do any potentially incorrect assumptions, it would not enforce the-one-versioning-policy-to-rule-them-all ... simply, it would be more flexible

Point is:

(a)library client MAY NOT know what future versions he will be compatible with, but
(b)library provider DOES know if he breaks anything from the past or not…

You could have versioning sequence like this (trying to make my point, not to be factually exact):

  1. kestrel
  2. merlin (backward-compatible=false)
  3. hopper
  4. mantis
  5. tiger (backward-compatible=false)
  6. 6.0 (backward-compatible=false)

(legend)

Version 7.0 might break your app or might not… You don't know TODAY. Developers of 7.0 WILL know better… TOMORROW

Your dependency could read like:

  • depends: merlin (ignore:mantis) #mantis has some known regression bug

Module system gives you either Merlin, Hopper, Tiger or 6.0.

When 7.0 will be released, you might end up running under 7.0 as well, if it WILL be declared compatible… It not (backward-compatible=false), 7.0 won't get into your way…

I would LOVE to know what is wrong whit THIS approach… Because to me this is more sound a solution than playing games with numbers...

Related Topics >>

Comments

I guess I'm not concerned about releasing a 2.0 version of a product and being able to indicate it is backwards compatible. If there is reason enough to release it as 2.0 then either it is no longer compatible or there is something worth the client knowing about (even if that thing is only that I worked my ass off to make a bunch of improvements). In which case, I want the client to have to explicitly consider 2.0 as a separate entity than 1.0 because of compatibility issues or just to force them to acknowledge the work I had done. If I wanted seamless upgrades, I would have just released it as 1.1.0 and called it good. Just because you don't want to follow the rules of the numerical schemes doesn't mean it is wrong or that it sucks. I could do the exact same thing under your system by refusing to mark things as backwards-compatible=false. The client would have unexpected behavior but that doesn't mean that the version scheme sucks, it just means that I as a provider am an idiot or really hate my users.

Maybe you don't want to control granularity but I sure as hell do. I want to be able to decide whether I only want bugfixes (1.1.X) or if I want to take a risk and try the new version that promises major performance improvements but hasn't been as widely tested as the rest of the library (1.X). A performance improvement isn't a backwards incompatibility so under your system it wouldn't be marked as such but there is some risk involved. So maybe you live in a world where libraries are static and have full test suites so they can confidently encode compatibility with a single bit, but this is reality knocking and that isn't the case where I'm sitting.

How did we get from module systems to build systems back to module systems? OSGi lets you specify either a dynamic dependency with ranges or lets you specify an exact version, so on that front it does do exactly what you say isn't done. In terms of build system, well I believe you can specify version ranges to maven so it can resolve dependencies, so I guess there is a build system that does exactly what you said isn't done.

Indicating compatibility doesn't change what you can call stuff under your system but you're wrong about not being able to use versioning to indicate the product's value with a numerical scheme. In the first paragraph I show an example--using 2.0 to indicate that something special was done. 2.0 may be compatible with 1.0 and then I have to make a choice as a provider: do I care about my clients and compatibility or do I care about my ego? In the former case, I would just call it 1.1 to indicate compatibility. If I care more about my ego, then I'll call it 2.0 and force my clients to at least evaluate the new version so they can see all the new whiz-bang features that I did. Why do I need to be able to call something "wow_I_cannot_believe_how_awesome_this_version_is_and_how_awesome_I_am, backwards-compatible=true"?

Verifiable - depends on who you are. For a module provider there is no *future* because once it is released it exists. From the client, there is a future and it is considered under both systems. Your system doesn't remove the future at all. Under yours there is a future that is indicated by a version tree that each entry has a backwards compatibility flag. Under the numerical system, there is a larger version number. As I've stated in the past, I gain nothing (no compatibility assurances, no more peace of mind) under your system that I don't have under the numerical systems as long as the module provider is following the rules of whichever scheme he is using. If a provider makes an incompatible change but only increments the service number, then things will break. If a provider makes an incompatible change but doesn't mark it as backwards-compatible=false, then things will break. Without contracts or comprehensive test suites, there's an element of trust required under either system. So it simply is a philosophical debate between how to encode compatibility and how many bits are needed to encode that information.

"we might be interpreting past incorrectly but we still have better chance to judge past correctly than predict correctly the future"

I agree that I can interpret compatibility for the past (e.g. the version that I created the app with) than I can for any other version that I haven't explicitly tested. But by your own admission, we still can't make any more guarantees about compatibility under your system than a numerical system for any version other than one we've explicitly tested.

The only way a module system can reason about compatibility is if each client provided a test suite for a dependent library to ensure compatibility or through the use of contract-based programming that could be verified programmatically by the module system. Then it places the burden on the client to test everything they need to work correctly, instead of burdening the library provider. I as a client have no one to blame but myself if an dependency upgrade breaks my app. However, I think we can both agree that both of these options are unrealistic and not ever likely to happen. Anything less than this approach, though, means we have to make compromises in the ability to reason about the compatibility of the library. Given that compromise, why should we adopt a whole new system that can't give us any more assurances than the existing, working solution?

The version history is useful if it included a more robust model for classifying the type of changes in each new version. But even then, it must be recognized that it offers us no more assurance that a specific version will break our app than using the conventions encoded in numerical versions. A library making rigorous use of your system will do no better than a library making rigorous use of numerical versioning conventions when the module system has to reason about dependency resolution.

Hmm. But two dependencies could still happen to request incompatible transitive dependencies no matter what. If the two dependencies don't pass objects back and forth, that's OK. But presumably that would happen a lot. I think any answer would be related to the bytecode hotswap question and the general question of static vs. dynamic typing. Also schema/service versioning. Hmm. Tough problem.

@mthornton

Hmm, I am convinced that I have answered your question. Maybe I don't understand what you mean or I am missing some detail...

Yes, you specify dependency as Servlet-2.3 or alike in your module. No asterisk, no wildcards, exact version you like.

Your build time dependency is satisfied when module system provides you with Servlet 2.3 API module, or maybe 2.4, 2.5 or whatever most recent version is available and is declared compatible... This should be enough if this is regular WAR to be deployed into servlet container...

Now image you are building standalone webapp using embedded Jetty. You prepare runtime module with the launcher class providing main() method which sets up and configures Jetty server with your webapp... In this runtime module, you would need to define all your runtime implementation choices: Jetty, ActiveMQ, Hibernate, etc

If you don't provide runtime dependencies, your imaginary service API call getService(MyServiceInterface.class) will fail to find any implementation (e.g. JPA infrastructure will not find Hibernate).

And this is the point where you can ask what a module system could do for you and how? The only thing it could do is check at RUNTIME if there is API module in your dependencies for which there is no implementation available - and maybe issue a warning or so...

The question is whether this behaviour is reasonable? Whether the module system can make any such decision robustly enough?

To be honest, I seriously doubt so. Without extensive contextual knowledge of your architecture, all a module system can do is a wild worthless guess...

But this is way ahead of the current question: can we use version history tree for more reliable dependency resolution?

I believe so. :-)

I was thinking of it from the point of view of the module writer --- what do I put in my module declarations. Something like servlet 2.3* for example. I don't care if it is provided by Jetty, TomCat or GlassFish, but the module system ought to check that my constraint is satisfied even if the application assembly is manual.

@mthornton

If you mean automatic selection of the suitable *implementation* for the *APIs* you depend on, then short answer is: it would be nice but it won't work well enough

The ideal situation is as follows:

  1. there is separately developed and versioned API (like Servlets 2.3)
  2. there are one or more implementations of the given APIs.
  3. for implementations to be pluggable, your library/module codebase is developed against the APIs. You are unaware of the implementations. *This is a MUST*
  4. When application is finally assemblied (something I call runtime module), suitable implementations need to be selected for APIs being used. *But only at this point, not sooner*

Now, automatic selection of the suitable implementations by the module system is possible, given enough metadata in module repository.
Personally, I am not sure how to make this useful.

When building *runtime module*, application assembler knows better than module system, that he wants to run this thing under Jetty6 (servlet API) with ActiveMQ (JMS API) with Hibernate backend (JPA API)
I cannot imagine module system that would be capable of making such a decision. Most that it could do is emit a warning like "Using API JMS-1.x" but no implementation is found in runtime module dependendencies. App may not work..."

However, I think this is a little bit beyond the scope of this proposal, at least in the first iteration.

Ad. java.lang.Package:

  1. My understanding is that Specification-Version means "I implement this spec defined elsewhere" (while Implementation-Version means what you expect)
  2. Also , I believe module system will deprecate package versions.

    Isn't it reasonable to get automatic resolution even if we have only specified a required specification (or range)? Specifications do get revised and sometimes are not backward compatible. Note that the jar specification has always included tags for both specification version and implementation versions. These are also visible in the java.lang.Package class (which specifies yet another version number syntax --- arbitrary number of numeric components).

    @tompalmer

    But two dependencies could still happen to request incompatible transitive dependencies no matter what That's why you should always specify your dependency as a minimum requirement (not the strict 1.2.3.* and nothing else) so that the module system could select freely anything that is declared compatible. If the two dependencies don't pass objects back and forth, that's OK Exactly. If you use your dependency (like some CSV parser) only internally, you know it. You let the module system know about it (you mark it *internal* or something like that). And the module system will know that if there is a version conflict, it could be resolved by module isolation You will end up with two or more incompatible versions of the module, both used internally by their respective "owners" and not accessible to anyone else, thus avoiding the conflict to manifest

    Of course, an unresolvable conflict may occur and the job of the module system is to detect and report it. If this happens, you need to resolve conflict manually either (a) by using dependencies that do not have conflicting requirements, or (b) by explicitly enforcing your solution, whatever it is. Module system should always provide the means to enforce anything. In this case, it is up to you to know what you are doing...

    But the morale of this story is:

    • require-the-least: always define minimum dependencies to maximize your interoperability potential
    • minimize-the-exports: always utilize as many of your dependencies as possible internally to avoid conflicts or enable coexistence of conflicting dependencies
    Anyway, you are right: it is not easy to design this properly. Tough call, indeed. But I believe it is both needed and possible. That's why we design the module system...

    You may very well be right, however I find it telling that almost everyone has standardized on numbers (for better or for worse).

    You didn't answer my question regarding how versions are ordered. How is the version history tree represented? And can I reason about what is a newer version? What sort of granularity can I reason about? How do you interact with other version schemes? At some point you're going to have to work with someone else who isn't using your scheme, so how do you build a version history tree from their numbers? You have an interesting theory. But at this point it is just that--a theory. Until you've released software under your system and proven the merits of it, it will only remain a theory.

    It seems like version schemes are like editor choice--it's highly personal and you're never going to please everyone. However, I'd much rather go with the tried and true approach rather than something completely new and untested. Furthermore, I think version schemes are small potatoes compared to the original context--the merits of the proposed module system and its ability to inter-operate with other existing systems.

    Guys, please,

    it is not important what the default would be! compatible or not...

    it is not important how the versioning looks... numbers or codenames...

    what IS important is that (1) everybody can choose his own versioning system and policy that fits his particular needs, and (2) module system does not need to be aware of this particular policy

    The question is: can we replace implicit global versioning rules defined and enforced by module system and BY VERSION HISTORY TREE with arbitrary and explicit compatibility declarations?

    How do I specify that any implementation of Servlet 2.3 is acceptable except GruSomeInc 1.4? This is easy when there is only one provider of a given specification, but many specifications have implementations from a number of organisations.

    Got one more thing to say to all of you:

    Versioning is crucial for module system. It is not just funny label on a jar file. It is mandatory for reliable conflict resultion made by module system.

    Module system simply CANNOT make any reliable decision with built-in number-magic...

    While releasing 0.1 after 3.0 is ugly, (1) it would be well documented in version history and (2) it would just work (because if 0.1 released after 3.0 is compatible with it, who cares?)

    Don't be scared about freedom:-) You're not gonna nail your window so you couldn't jump outta it, right? :-)

    @jareed

    Well, finally. I like your argumentation. And you've made a point.... which I have accepted, have thought about, and now I am going to debunk it :-)

    You say: "The best person to decide whether a library is backward compatible or not is me."

    Well, my point is you cannot tell TODAY whether NEXT library update (TOMORROW) won't break you code, not even on 1.0.0.X level.

    And we're not talking about final applications (they always, in any module system, should have the right to make final decision about. what versions they are using...)
    We are talking about generic libraries that are expected to be mixed with each other. You cannot just fix your library to JDK 1.4.x (unless you have a very god reason to)

    Imagine you have like 4 dependencies in you library and you restrict every of them to some 1.2.X release...
    Imagine there are 4 such libraries (with same restricted dependencies) and you are going to use them together in your next library (or application).

    I dare to say, you are never gonna make these 4 libraries work together because their restrictive most likely conflicting dependencies...

    Now suppose you have hybrid system: version history tree PLUS loosely defined non-restrictive number-based versioning metodology. Because everybode will keep using the numeric versioning... It is here for a reason... You have the best of the both worlds. What can we lose?

    I am not against the numeric versioning. I use it too :-) I am against module system caring too much about what my versioning looks like and what it does mean...

    I'm with Patrik. Let's not try to force someone's, anyone's, idea of a good system on the entire planet.

    Not only isn't it going to work, it's an insult to the intellect of everyone else out there to consider yourself to be so much smarter than them that you can tell them that yours is the One Way.

    You are on the right track to understanding what I really meant. Congrats. :-)

    My proposal is far from complete. Never intended to be.

    All I wanted is to propagate the idea. Show that there IS a better way. It is a fundamental shift, change of paradigm. That's what is hard about it. People have dificulties thinking in new models they are not used to.

    Once we accept this as a good (better) idea, we can elaborate on it, craft the model properties to an acceptable and usable level... (just like you did with bug-fix, behaviour-change, etc - all the stuff I intentionally ommitted just to highlight the idea)

    I don't accept your RoR example / association.
    Infering library compatibility based on simple numeric versioning scheme is seriously flawed. Is is just all totally wrong. (Why try to predict the future if you can just look back with much more certainty.) I tried to point this out. And I think I've said enough.

    Integration with other/existing module systems was never my concern. Personaly (and this is a warning), I think this approach (make the new beast compatible with old beasts) will be a serious limitation if not even cause of the failure...

    Just think about it.

    @jareed

    About being condescending: I am sorry, I didn't mean it that way. Maybe that's just my English.

    I am dismissing your RoR example not because it is invalid but because it is irrelevant.

    My system would not add extra work on shoulders of providers: they already have to check its product for backward compatibility. If they don't care, they can just tag it incompatible by default and they are done.

    My system would not add any extra work on you as a user because in ideal case you don't have to do anything beyond specifying one version number - the LOWEST / EARLIEST one that by its specification fits your needs. You don't have to wonder what other version are available, whether there is some major release available that you might wanna use, etc...

    My system takes the burden from 1000 of users of a particular library AND gives this responsibility to 1 single provider of the library. So it actually saves the work, right?

    My claim is that we cannot predict future but we can look back. Yes, we might be interpreting past incorrectly but we still have better chance to judge past correctly than predict correctly the future. And I am pretty confident that this claim is undeniable :-)

    As for integration (personal confession): yes, I am aware of the marketing problems. But then, I've been always better in technology that in marketing. Striving for marketing, I risk ending up with technology that sucks supported by marketing that sucks. So, that is my choice, at least for now: do what I can do best: technology.

    @tompalmer

    I believe there is a misunderstanding about what the module system is (or should be). (Edited: trying to be more polite :-) )

    Module system has very, very difficult task of gathering all the required dependencies of the module and perform conflict resolution.

    If my library A depends on log4j 1.2.1
    and your library B depends on log4j 1.2.18
    and somebody attempts to use both our libraries in his project, with strict hash-md5-sh1-verified dependencies he gets stuck.

    And what is he gonna do? Manually resolve all the dependencies?
    What is module system gonna do? Accept whatever the developer says or is it gonna complain that log4j 2.0.0 (which developer selected instead of those 1.2.x versions) is not compatible and the software won't work? What is the purpose of such module system after all? Just politely inform developer "I might need something called log4j 1.2.12, please tell me what to do since I am so hopeless..."

    Man, that would really suck.
    There is a lot more to module system than just getting the bunch of jars files at your doorstep.

    I completely agree with your statement that compatibility metadata needs to be addressed. So instead of saying "you can call the version whatever you want, it's a better system", lets start addressing the real issue--modeling the compatibility requirements. That's what needs the attention, not what you call the version. Once we have a workable compatibility system, it can be retrofitted onto a numerical system (in terms of extra metadata, effectively making the numbers useless for reasoning) or can be incorporated into a new style of assigning versions like yours. Well, that is exactly what I am saying the whole time: My own quotes from this discussion: My proposal is far from complete. Never intended to be.

    All I wanted is to propagate the idea. Show that there IS a better way. It is a fundamental shift, change of paradigm. That's what is hard about it. People have dificulties thinking in new models they are not used to.

    ...

    Beyond what was said, I believe that versioning should not be confused with compatibility.

    Module system should care about compatibility, not versioning. Encoding compatibility metadata in versioning scheme is a good hack but it's just plain wrong.

    Call it loose coupling, separation of concerns, whatever you like.

    I guess, after all, we might happen to agree to agree :-))

    Your argument about USES, IMPLEMENTS, EXTENDS is intriguing. You don't handle this situation any better under your system since you only have a single backwards-compatible=false. So you'd have to mark it as not backwards compatible and force both the implementer and client to evaluate it before using it as a dependency which I could do in the numerical scheme by bumping the major version number. Doesn't changing an interface break backwards compatibility anyways? I think the Eclipse guys always just created a new interface with the new methods. Then they could implement both interfaces and keep backwards compatibility while also allowing them to evolve the API. This allowed old clients/implementers to continue working will allowing new clients/implementers to take advantage of the new features. I agree that it doesn't look very nice, e.g. a LogManager and a LogManager2 interface, but it's up to you as the provider of whether you want to maintain backwards compatibility or force updates. Long story short, the scenario is no better handled under your scheme or the numerical scheme.

    I completely agree with your statement that compatibility metadata needs to be addressed. So instead of saying "you can call the version whatever you want, it's a better system", lets start addressing the real issue--modeling the compatibility requirements. That's what needs the attention, not what you call the version. Once we have a workable compatibility system, it can be retrofitted onto a numerical system (in terms of extra metadata, effectively making the numbers useless for reasoning) or can be incorporated into a new style of assigning versions like yours.

    so, jareed, to avoid loss in my own verbosity, please:
    What do you think about my previous api/implementation/client example? I would like to know...

    thnx

    Also please note an update of the previous comment (ad. verifiable). thnx

    Sounds like a plan. I won't try to poke holes in it anymore. :)

    Yes, I see that. Something went wrong and we followed the wrong path in the discussion... Sorry about that. :-(

    But we got there eventually :-)

    I have read again some highlights from my original blog post (2006) and this discussion and I realized I have been doing *really* bad explaining my ideas (at least related to this subject).

    Maybe it would be worth putting all this stuff together (with all the crap removed) and try again, with more focus on compatibility metadata, work out the details little more, and dealing properly with all the stuff that caused confusion here. (Again, I see examples worked more that words.)

    What do you think?

    So maybe instead of spending too much time caring about what a version looks like, we should be looking for a way to indicate the scope of change/likelihood that a particular version will break with an update. I don't think just "backwards-compatible" is enough to capture it. But if versions could be tagged with some combination of bug-fix, behavior-change, feature-enhancement, backwards-incompatible then the module system could use that information to find the most appropriate candidate to fill a dependency. E.g. I could say I depend on FooBarLib 1.2.3.4 and will accept any later versions tagged as bug-fix or feature-enhancement but not behavior-change or backwards-incompatible. Then the module system could look at all other dependencies on FooBarLib and hopefully find one version that satisfies everyones dependencies.

    I was there back at comment 7 or 8 as well, but your intervening arguments haven't been focused on developing compatibility metadata.

    I think the advantage with numbers is that one can sometimes reason with them. If I saw your group of versions up there, I have no idea of how to order them. Where does it say that mantis is after merlin? Somehow you have to order them, so at some point numbers come into play.

    I like the ability for the library provider to say this is not backwards compatible as a pointer to clients. But as a client, I can't always trust the provider because they may not have a full suite of tests that exercise all of the nitty gritty areas of the API that I use. So ultimately the client has to make the decision of what is backwards compatible or not and whether to allow new versions. Using numbers at least I can say I need at least 1.2.3 and will accept anything in the 1.2.X where X > 3. How would you encode that information into your system?

    Even under your system, I would suspect that 95% of the folks would still use numbers for identifying versions. Your system provides the ultimate flexibility but sacrifices the ability to somewhat reason about the versions. I'm scared of a versioning system that would allow me to release a 0.1 after a 3.0 as a perfectly valid release schedule.

    I'd default to "not compatible", but that makes some sense. Going further, my own personal opinion is to always assume incompatibility for most libraries. Generally, it's better to download another 300 KB (or even 3 MB) than to have broken software. Just SHA-1 the bundle, and there's your version number. Just my own opinion. (The core JRE could even download alternative versions of itself if it were cool enough. Can Java Kernel do this?)

    First of all, I thank you for your time you are still investing into this discussion. I really appreciate that.

    Second: sorry for being too verbose. I just cannot do this any shorter, and I really try :-( I want to be able to decide whether I only want bugfixes (1.1.X) or if I want to take a risk and try the new version that promises major performance improvements but hasn't been as widely tested as the rest of the library (1.X) So now the version number also indicates how well or widely tested the library is?
    Man, you want to encode hell of a lot of information in those 3 digits… How did we get from module systems to build systems back to module systems? Sorry, my bad. Paragraph "There are only two modes of operation in build system..." should read "...module system…"

    Confusion caused by the fact that I believe module system and build system designs MUST be highly integrated… But that is another story...

    About things being done or not done:
    Ad OSGi/Maven: yes you can specify ranges, absolute value etc in your dependencies… But it is limiting it terms of future development or future interoperability
    What I had in mind is the notion of persistent dependency graph, persisted at build time or at runtime (as your prefer), so that next time when you build or run your application, it will build or run with the same set of dependencies. And this information is not part of your dependency declarations, it is persisted aside from them .
    I.e. You should not freeze dependencies in your library as it limits its integration potential. But as an application assembler, you you still want reproducible builds and reproducible runtime…

    And this is not addressed in OSGi nor Maven. 2.0 may be compatible with 1.0 and then I have to make a choice as a provider: do I care about my clients and compatibility or do I care about my ego Maybe.
    And maybe marketing guys would explain to you that 2.0 would sell better than 1.1.
    Or maybe you as a developer want to be able to tell how much work have been done between the versions just by looking at the version sequence…
    And maybe you end up using two versioning schemes: one for the module system, one for the public relations.
    That is reality knocking, too.

    Ad verifiable:
    (This section was removed because YOU ARE basically RIGHT, following your logic. There's no need to say more. My argument is only an additional motivation for a change, only partially valid, and not by itself undeniable reason. I.e. had it been the only argument I've got, I would have nothing. So yes, considering this issue in isolation, you are right and I was wrong.)

    So it simply is a philosophical debate between how to encode compatibility and how many bits are needed to encode that information. Yes and no. My another claim (that is hopefully supported enough in my previous comment) is that required compatibility metadata are so complex and heterogenous that you need something more robust, more extensible and more human readable than just version numbers.

    I have written some response to what you wrote, but I also want to discuss following interesting scenario.

    So, to avoid long-long mixed posts, let me first discuss the scenario I had in mind:

    My claim is that simple version number is not flexible enough to clasify dependecies.

    Suppose you have an interface in your library/module "api" version 1.0.

    interface LoginManager { void login(String username, String password); } Now, you add support for some sort of security certificates:

    interface LoginManager { void login(String username, String password); void login(String username, String password, Certificate cert); } What would be the version number? 2.0 or 1.1?

    Is it backward compatible?

    One might think IT IS:
    I am just USING the interface: looking up an object from some kind of service registry and calling login(username,password) on it. I don't care if there's another method available because I don't use it.

    One might think IT IS NOT:
    I am IMPLEMENTING the interface. If the interface is extended with additional method, I need to implement it, too.

    So imagine you have three standalone modules:

    • api 1.0
    • implementation 1.0 (depends: api 1.*)
    • client 1.0 (depends: api 1.*)

    Now you add this new method into the interface and what happens?

    1. implementation is broken whether the version is 1.1 or 2.0
    2. client should work well with the change whether it is version 1.1 or 2.0

    So, is new "api" compatible or not?
    From the "implementation" view, it is not.
    From the "client" point of view, it is.

    How do you encode this in single version number?

    Maybe you need additional metadata to clasify your dependencies:

    • api 1.1
    • implementation 1.0 (implements: api 1.1)
    • client 1.0 (uses: api 1.0)

    There is a difference between USES, IMPLEMENTS, EXTENDS, etc And we also need this system to be extensible. So that if we find use case and semantics for PROVIDES, we could add it to the model.

    With numeric versioning, system is CLOSED and scales in one dimension only.

    Maybe this sounds too complex. But why not to think about it?

    So maybe instead of spending too much time caring about what a version looks like, we should be looking for a way to indicate the scope of change/likelihood that a particular version will break with an update. I don't think just "backwards-compatible" is enough to capture it. But if versions could be tagged with some combination of bug-fix, behavior-change, feature-enhancement, backwards-incompatible then the module system could use that information to find the most appropriate candidate to fill a dependency. E.g. I could say I depend on FooBarLib 1.2.3.4 and will accept any later versions tagged as bug-fix or feature-enhancement but not behavior-change or backwards-incompatible. Then the module system could look at all other dependencies on FooBarLib and hopefully find one version that satisfies everyones dependencies.

    This allows for more flexibility but requires the library providers to be more rigorous in their tagging of versions. I personally don't really see the added benefit that the freedom gives me over the extra work. It's a lot like the old convention over configuration debate. Using something like RoR you give up the flexibility of being able to structure things the way you want but you gain the ability to get more done quicker. Under your system you don't have the inflexibility of a numbered version system but you have to do more work to specify what kind of change it is and guess at how it will affect your clients.

    And in the end, you can tag your code and release it under whatever version you want to call it. But if you want it to work with the existing and proposed module systems, you or someone who wants to use your library is going to have to map it to a number scheme anyways.

    I like your example. As I said before, I'm not against the idea, I just wanted more information. I'm still a bit skeptical because, quite frankly, I don't trust the upstream provider to completely test things and mark it as backward-compatible=false for my application. The best person to decide whether a library is backward compatible or not is me. Now you can argue the same for existing number schemes--that you can't trust the provider to test things to make sure they aren't breaking something. However, the advantage of the existing system and what is lacking in your system is no way to assess how big, how likely an update is to break my application. Under number-based systems I can decide to be conservative and only allow updates in the 1.2.3.X range, or if I'm feeling more adventurous, I can allow updates in the 1.2.X.X range. Under your system, as you pointed out in your example, all changes are equal because you can't infer anything from the version. There's no way to gauge how large of a change it is and therefore how likely it is to break something.

    I'm not against freedom. Your analogy is flawed; I'm not going to nail my window shut so I can't jump out of it because I trust myself. What I'm not going to do is let you 'upgrade' my window and listen to you when you tell me "go ahead and jump at the window, this 'merlin' window will stop you because it's backwards compatible with the previous 'mantis' window you had." :-)

    The other flaw in your argument is your statement: "Prove me wrong." It's not mine or anyone elses responsibility to prove every crazy idea you have wrong. On the contrary, it's your responsibility as the originator of something new to prove it correct to enough people to gain momentum or acceptance. You haven't convinced me yet.

    Beyond what was said, I believe that versioning should not be confused with compatibility.

    Module system should care about compatibility, not versioning. Encoding compatibility metadata in versioning scheme is a good hack but it's just plain wrong.

    Call it loose coupling, separation of concerns, whatever you like.

    I just cannot release version 2.0 that is backward compatible with version 1.0.
    That sucks.

    Even if I could, there's no way for you to get updated to backward-compatible 2.0 automatically. You need to be aware of the 2.0 version (which you can't be: it's the future).
    That sucks.

    Module clients are confused (by being able to play with the version numbers) the same way as you or tompalmer: that a more strict dependency requirement means more safety for you and that you have to decide...
    That sucks.

    Clients are confused to think they want to control any dependency granularity. In fact, they don't. You just either have compatible dependency or you don't. No number games.
    That sucks.

    There are only two modes of operation in build system: (1) you either are dynamic and let the system decide at its will what version is right, or (2) you freeze all the versions at build time and never let the system do any fancy tricks (requirement of the reproducible release builds)... People do not realize that and no module system I am aware of supports this...
    That sucks.

    System forces me to use versioning to indicate compatiblity level. I cannot use versioning to indicate the product's value (work done, resources invested, change amount, line of code, performance improvements, whatever the metrics are).
    That sucks.

    ad. VERIFIABLE: You can verify that *current* version is compatible with *previous* version. You cannot verify that a *future* version is compatible with *current* version. Simple as that. No need to get started about the verification process (unless you have time machine)

    Anyway, I agree that when you accept the given version conflict resolution model and limitation it enforces (which I don't), the OSGi semantics are defined pretty good (simple enough yet consistent). JSR277 just sucks...

    Peter's quote is impresive, I like that very much. We may agree to disagree, why not. I don't even need to have the last word :-)

    Patrick, both OSGi and Eclipse (being built on OSGi) have very well defined semantics for each numerical component in their version scheme, and the major (first) component is the backward compatibility indicator. But beyond that, it offers a granularity that your system doesn't because it lets me decide whether I only want bug fixes or if I am willing to accept newer versions with "externally visible changes" or if I care about backwards compatibility. It's up to me as the client as to whether I want to use only one bit of data (major component) or also to look at more data (the minor and service components). It also prescribes very well defined responsibilities of the module provider and very well defined rules for the module system.

    We don't need a new system because the OSGi version scheme already defines the responsibilities of the module provider, system, and client and has worked out the the semantics for how dependencies are resolved.

    Verifiable in what sense? No test suite is being run against a library to ensure it is backwards compatible so I just have a single bit of data to verify against--whether the library provider thinks the new version is backwards compatible or not and whether I trust their assessment. That's no more verifiable than what already exists, nor does it give me any more data to work with when deciding on what version to use than incrementing a numerical component. On the contrary, I actually have more data at my disposal when analyzing a potential dependency resolution because I have 3 components to work with.

    Given a reasonable level of trust of the provider and the system, which is also required by your system, I, as a module client, can expect to have my dependencies resolved in a predictable way, and at a granularity not possible under your scheme.

    I think Peter Kriens put it best: With an infinite number of builds or variations, 9903520300447984150353281023 possible bug fixes (80.000 fixes per microsecond until the Sun implodes), 4611686014132420609 backward compatible changes, and 2147483647 incompatible releases, the OSGi spec seems to have enough room to breathe for mere mortals.

    We're just going to have to agree to disagree.

    @jareed

    If the numerical system would be as simple as "major (first) version number is backward compatibility indicator, others don't matter" and your dependency declaration would be as simple as defining this major number, then, and only then, you would be right.

    I am sorry but this is not the case. Proposed versioning is complex, enforce bad practice of requiring too specific version (1.2.18.* and alike) and has other issues I already talked about.

    My points are simple:

    I, module provider, define this API and declare this contract. I, module provider, am responsible for verifying that my product fulfils this contract. Period.

    Anything beyond that, any client assumptions about the behaviour explicitly not specified in the contract, is beyond the responsibility of me, the module provider, or him, the module system.

    I, module system, do not implement any explicit or implicit logic to determine compatibility fo the various libraries, instead, I rely entirely metadata provided with the given module by the module provider.

    For situations where anything goes wrong for the client, I ,the module system, provide the means to enforce whatever the client thinks is more appropriate

    What is wrong with that?

    why should we adopt a whole new system that can't give us any more assurances than the existing, working solution? It gives us more, because it defines clear and well assigned responsibilities.
    It promises more because it depends on verifiable instead of just predicted data.
    It is the right time to think about this because we are designing the new shiny Java Module System (R) JSR#277.

    Each new release declares (1) its version, (2) the version it is derived from (and all its history) and (3) compatibility with the previous version

    Simplified straightforward example:

    Your library has something like META-INF/VERSION.MF which says: Version: 1.0 History: After glorious success of version 1.0, you release a bugfix release: Version: 1.0.1 History: 1.0 And another: Version: 1.0.2 History: 1.0.1; 1.0 Now you finally implement that minor small enhancement people wanted so badly: Version: 1.1 History: 1.0.2; 1.0.1; 1.0 For a long time since 1.0, you know you can do better API, better software, so you redesign APIs while reusing work you have done so far: Version: 2.0 (backward-compatible=false) History: 1.1; 1.0.2; 1.0.1; 1.0 While implementing new APIs, you realize that the whole internals of your software need to be rewritten from scratch... You do this while maintaining API compatibility. This is a huge rewrite, faster, better, more reliable... You haven't changed anything from your users' point of view (so it might as well be 2.1 or even 2.0.1). However, you think It deserves major version to let the clients know you did something big! Version: 3.0 History: 2.0 (backward-compatible=false); 1.1; 1.0.2; 1.0.1; 1.0 Syntax may be ugly or weird or unsatisfied... That's not the point right now...

    History is obvious. Every module keeps track of its branch history (single branch)
    Complete tree (all branches) can be assembled by merging history from multiple modules

    Yes, this is theory. Works well on paper. It is much simpler and more robust compared to Sun-optimized number-magic by JSR#277. Prove me wrong. At least in theory.

    I guess their proposal is already proved wrong. Does not work, not even in theory.

    People and companies use different versioning schemes (1.0#r1234, 2.3.b999, 10.0.2.1234-alpha-build8888, 20080530:2359, Windows-3.1, Windows-95, Windows-2000, ..., Windows XP, Windows 7)

    Freedom is power! Don't force the-only-version-scheme on us.

    @mthorton

    if you program against the APIs, like Servlet API, you depend on specification. In this particular case, there is only one specification a there'll never be another :-) because it just does not make any sense.

    Application Assembler or Integrator (or whatever we call the guy who decides to deploy your Servlet 2.3 based application) will have to make the decision whether to use GruSomelnc 1.4 or not.

    Your code, on module system level, does not depend on any particular implementation.

    Anyway, see my mantis example which is supposed to do exactly that: ignore known implementation that does not match expected specification (like: require JDK 1.4 or compatible, ignore 1.4.2 which has some regression issues)

    I was also thinking about the online repositories that would support some sort of live blacklists that could aggregate these kind of compatibility issues, and module system could utilize them when making dependency resolution. But right now, that's just an idea...

    No need to be condescending, Patrick. I don't have any vested interest in seeing your system fail so it's best not to alienate your potential users before you even have something to give me.

    You can dismiss my RoR example out of hand, but I can also dismiss your assertion that your system is better and is workable. RoR (and convention over configuration) exists and works. Where is your versioning system?

    You can no more look back any more certainty than I can look forward. As I have stated in past comments, I have no guarantee that the library provider has checked every case fully and is properly using backwards-compatible=false than I can trust the library provider to increment the 4th numerical component for only minor bugfixes. So in this case, knowing that the library has had 37 previous version does absolutely nothing for me. And the funny thing is, you're espousing freedom to call things whatever you want but to come up with a workable and viable system, you're going to have to impose convention (in the form of the wholly inadequate backwards-compatible or even in the more expanded set that I suggested)

    You can ignore integration with other/existing module systems, but you do this at your peril. You're not going to be release something on the world and have instant, widespread adoption.

    Maybe going a bit further on my original comment, I don't really even trust the library developer to know if it's backwards compatible. I think the simplest solution of all really is just for the system to require exact versioning based on content hashes. It's also the _simplest_ dependable system of all. (The sad case would be old software you want to use, but the old dependencies are no longer available. But maybe some kind of user override would be enough for odd cases like this. And anyone distributing software usually should distribute their own copies of dependencies, too, for default availability. That's the _only_ model for most web (JS) software today, anyway.)