Skip to main content

Fixing two problems with Maven + Mercurial + Hudson

Posted by fabriziogiudici on October 29, 2009 at 5:48 PM PDT

Today I've made some improvements with my Mercurial + Maven + Hudson
setup - and reached a new level of karma, being able to do automated
releases.



Let's go in order. First let me recap what happens with the Maven
release plugin (mvn release:prepare release:perform) and Mercurial:

  1. A check is performed that there are no uncommitted changes
    and a build is performed as a validity proof.
  2. All the version labels in the current project are updated
    (e.g. 1.3.7-SNAPSHOT to 1.3.7) and changes are committed
  3. A tag in the SCM repository is made for the new release
    (e.g. release-4.5)
  4. All the version labels are updated again (e.g. 1.3.7 to
    1.3.8-SNAPSHOT) and changes are committed again
  5. Another copy of Maven is spawned on a temporary directory
    where sources are checked out from the SCM, with the previously created
    tag; another build is performed and artifacts deployed.



It's a robust sequence, as it makes sure that there are no
inconsistencies in version sequences and the release is built exactly
from the tagged files.



But there are two specific problems with Mercurial:

  1. The fact that Mercurial is a distributed SCM is not fully
    exploited. You recall that commits are only performed locally and not
    shared until a push is performed. It would be a nice thing if the push
    was performed at the very end of the sequence, so in case something
    went wrong, you can rollback everything. Unfortunately, Maven performs
    the push at the end of step #4, and something can still fail during
    step #5. I suppose that since step #1 performs a full build, Maven
    developers thought that we have a proof of the validity of the build,
    but unfortunately in the real world everything can fail at every time
    (for instance, an OutOfMemoryError).
  2. The check out performed at step #5 is indeed a Mercurial
    clone, that is executed from
    the
    remote repository
    . This is a very stupid thing, since
    Mercurial
    repositories can be huge, so this operation can be time consuming
    and expensive; the local repository could be cloned in a more efficient
    way, but it sounds as a hard fix for Maven.



I'd say that the former is a real problem, while the latter is a
serious annoyance. Here how I fixed both problems.



First, I've used a property in the SCM section of the pom for
specifying the URL:



<scm> style="font-family: monospace;">
   
<connection>scm:hg:http://kenai.com/hg/forceten~src</connection>
style="font-family: monospace;">
   
<developerConnection>scm:hg: style="font-weight: bold;">${staging.hg.repo.url}
</developerConnection> style="font-family: monospace;">
   
<url>http://kenai.com/projects/forceten/sources/src/show</url>
style="font-family: monospace;">
</scm> style="font-family: monospace;">


<properties> style="font-family: monospace;">
   
<staging.hg.repo.url>https://kenai.com/hg/forceten~src</staging.hg.repo.url>
style="font-family: monospace;">
</properties>



In this way the URL can be overridden with a property such as  style="font-family: monospace;">-Dstaging.hg.repo.url=something:
the idea is to prepare a temporary, local repository where Maven pushes
all the changes made for the release; furthermore, the clone at step #5
is performed from that local repository, thus saving time and
bandwidth.



Second, I used the altDeploymentRepository
property of the deploy plugin (which is called by the release plugin)
that allows to override the definition of the target Maven repository
and make it point to a local folder. At the end of the process, I have
that all the
changes are stored locally
; in other words, I've used style="font-style: italic;">staging repositories
both for the sources and the build artifacts. If there are no errors, I
can later push the local Mercurial repository to the public Mercurial
repository and copy the artifacts in the Maven repository to
my public Maven repository.



This is the required configuraton for the maven release plugin:



<plugin>

   
<groupId>org.apache.maven.plugins</groupId>

   
<artifactId>maven-release-plugin</artifactId>

    <configuration>

       
<arguments>-Dstaging.hg.repo.url=${staging.hg.repo.url}
-DaltDeploymentRepository=release-repo-hudson::default::${staging.mvn.repo.url}
-Dstaging.mvn.repo.url=${staging.mvn.repo.url}</arguments>


    </configuration>

</plugin>



Basically we are propagating the relevant properties to
the spawned instance of Maven; there's a little verbosity as we need
the Maven staging repository expressed in form of a URL (see below)
while altDeploymentRepository needs
it prefixed with some specific Maven repository notation.



Maven should be called with these options:



-Dstaging.hg.repo.url=/my/hg-staging-repo
-Dstaging.mvn.repo.url=file:///my/mvn-staging-repo
style="font-family: monospace;">


For copying the Maven artifacts from the staging repository to a public
one the wagon-maven-plugin can be used, as it has an option to merge
two repositories (goal wagon:merge-maven-repos); the required
configuration is:



<plugin> style="font-family: monospace;">
   
<groupId>org.codehaus.mojo</groupId>
style="font-family: monospace;">
   
<artifactId>wagon-maven-plugin</artifactId>
style="font-family: monospace;">
   
<configuration>
style="font-family: monospace;">
       
<source>${staging.mvn.repo.url}</source>
style="font-family: monospace;">
       
<target>https://services.tidalwave.it/nexus/content/repositories/releases/</target>
style="font-family: monospace;">
       
<targetId>maven2-release-repository.tidalwave.it</targetId>
style="font-family: monospace;">
   
</configuration>
style="font-family: monospace;">
</plugin>



It is possible to call it by a separate Maven invocation after the main
build, or it can be
specified in the goals of the release plugin:



<plugin>

   
<groupId>org.apache.maven.plugins</groupId>

   
<artifactId>maven-release-plugin</artifactId>

    <configuration>

       
<goals>deploy wagon:merge-maven-repos</goals>

       
<arguments>-Dstaging.hg.repo.url=${staging.hg.repo.url}
-DaltDeploymentRepository=release-repo-hudson::default::${staging.mvn.repo.url}
-Dstaging.mvn.repo.url=${staging.mvn.repo.url}</arguments>

    </configuration>

</plugin>




In this way, mvn
release:prepare release:perform
does all but Mercurial
synchronization; the latter task just needs a regular hg push and can easily
be performed by a post-build script.



If I get an error, nothing gets out of my disk, so I can delete the
staging repos, fix the problem and retry.



It is also possible to skip the synchronization between the staging
repositories and the real ones, in case one only wants to test the
process. For instance, I'm trying to have the artifacts generated by
the assembly plugin deployed and I'm not able to do that yet; my trials
and errors can be done locally without polluting the public
repositories.

Related Topics >>

Comments

Excellent post on how to fix those two issues. Another way ...

Excellent post on how to fix those two issues. Another way of getting around it is mentioned deep in the mailing lists -
is to set two attributes

localCheckout to true
pushChanges to false

This will use the file protocol to copy over your local repository which will speed up things a lot and will disable the push option

Here are the details http://maven.apache.org/maven-release/maven-release-plugin/perform-mojo....

vandanagopal, you're right. This post is four years old, and ...

vandanagopal, you're right. This post is four years old, and unfortunately at that time those parameters didn't work properly - there was no other way that using those tricks. At present time they work, in fact some time ago I updated my SuperPOM so it uses them. I've still kept the capability of not having anything (hg changes and mvn artifacts) going out of the local disk - it's quite useful to wait and see that everything was fine before publishing everything.

BTW I've also updated things so they also work with Git - I don't use it, but some customers using my SuperPOM do.

The updated SuperPOM can be found at http://bitbucket.org/tidalwave/thesefoolishthings-superpom-src

Thanks for commenting and giving me the opportunity to update.

I'm having issues with this. Although your work around seems ...

I'm having issues with this. Although your work around seems to work I get a failure when trying to push:

Failed to execute goal org.apache.maven.plugins:maven-release-plugin:2.1:prepare (default-cli) on project Ouput-Handler: Unable to commit files
Provider message:
</pre>
<pre>EXECUTION FAILED

Execution of cmd : push failed with exit code: 255.
Working directory was:
Z:\NetBeans\output-handler
Your Hg installation seems to be valid and complete.
Hg version: 2.2.3 (OK)
</pre>
<pre>

Command output:
-> [Help 1]

I believe that is missing credentials to do the push. Any idea?

Generally speaking, try to run mvn with the -X option that ...

Generally speaking, try to run mvn with the -X option that provides more information.

Specifically to the problem, yes, it can be a matter of credentials. If your hg is always asking for a username/password when you use it (try it directly from the command line), the thing will always fail because mvn can't provide the credentials. You should read the hg manual in order to configure $HOME/.hgrc with the proper credentials and make sure that whenever you use it from the command line it doesn't prompt. A better solution, even for intrinsic security, would be to use SSH certificates. I've experience with Linux Ubuntu and Mac OS X: if properly configured the system should ask for the certificate password only once per session.

Nice post! I will adapt to

Nice post! I will adapt to git in my projects.

Issue with Maven release plugin

Hi, I followed the above article and implemented the suggested steps in pom.xml. When I run the following goal on the project from command prompt: mvn -e package -Dresume=false release:prepare release:perform Its giving the following exception: Caused by: org.apache.maven.plugin.MojoFailureException: Unable to commit files Provider message: EXECUTION FAILED Execution of cmd : push failed with exit code: 255. Working directory was: /home/test/.../../../ Your Hg installation seems to be valid and complete. Hg version: 1.1.2 (OK) Command output: at org.apache.maven.plugins.release.PrepareReleaseMojo.execute(PrepareReleaseMojo.java:169) at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694) ... 17 more Did I missed any steps? How to solve the above issue? -Sravya

One unpleasant stuff with the

One unpleasant stuff with the Maven/Hg integration is that it only gives partial diagnostics when a hg command fails. I suggest to reproduce the error and, as soon as it fails, try to manually run from the command line the hg push command exactly in the same way as Maven tried. So you can get a better information about the failure. For me, the most common cause of failure is when there's an extra head, and hg refuses to push unless you specify the -f flag. This can be handled in two ways: either you seldom create extra heads, and in this case you manually review them, manually perform hg push -f and then re-launch Maven; or you specify the -f flag as a default for the hg push command (can be done in $HOME/.hgrc, see the hg manual).

reply

Hi fabriziogiudici, Thanks for the quick reply. I tried to run the command 'hg push' manually from the commandline after the failure. it worked with out any failures, verion is changed in pom.xml so only that changes in that file are commited. Actually when running the command "mvn -e package -Dresume=false release:prepare release:perform" , it promted for release version of project, release label of tag etc. eventhough the command exection fails , its pushing the changeset (changes in pom.xml) to the default branch in repo. but no tags have been created under tags.

same on Windows and altDeploymentRepository

Hi, thanks for your blog about this, it really helped us to look into the right directions. I do have some remarks, and hope that you still notice that there's a reply to such an old blog post and can answer.
  1. It took me some time to figure out, but the staging.hg.repo.url should NOT start with file:// or file:///. You are doing correct, but there are other examples out there which only work on Linux. They tell to use scm:hg:file:///${basedir} but doing so on Windows results in an error that the repository /C:\... cannot be found. If you omit one slash and just have scm:hg:file://${basedir} the scm provider complains. This is actually an unfixed bug: http://jira.codehaus.org/browse/SCM-508. Just have an scm:hg:${basedir} is fine
  2. When you deploy the artifacts to an alternative local repository, you are fixing the release plugin in general, not just for mercurial usage, right? Because when you use Subversion, you're into the same dilemma. First the commit is happening, and then you deploy the artefacts into the repository. Or put the other way round: if you could live with the fact when using Subversion, you could also use the release plugin the same way with mercurial.
  3. Have you found any way to do the final hg push as part of the release:perform?

Hi. That f**ling c:/...

Hi.

  1. That f**ling c:/... :-)
  2. Yes, it's a problem in general, but my trick won't work with Subversion - or, at least there are some details that I wouldn't know how to manage. The good thing, in this scenario, about Mercurial is push: and Maven when does a release executes a hg push, and it's easy to tell that the target is a new, empty repository. How would you manage this with Subversion? You could even create a Svn repo on the fly, but I wouldn't know how to commit to it. Perhaps by executing a "relocate", but I'm no expert with that.
  3. No. In the end my whole procedure still relies on running Maven twice, with the second run taking care of the final push.

Thanks for replying :)

Hi Fabrizio,
  1. C:\ not C:/ ;) but it sucks either way ...
  2. Now I understand. I thought you blogged this because it's a unique problem with Mercurial combined with the maven-release-plugin. But it's a inherent problem of the maven-release-plugin for which only a solution with mercurial exists. What do you think about using ${basedir} as the repository? In combination with the localCheckout parameter of the release:prepare goal that works quite well.

Subrepositories

Would Mercurial Subrepositories help?

A very neat trick. I've hit

A very neat trick.

I've hit this problem several times in the past, but it gets worse with a complex project - which I have two: One has modules which have to be built on various platforms (native code) and the other retrieves xsd source from a remote svn repository which goes down every so often.

In either case, using a staging repo ensures that the public repo only gets the change once the release build has worked completely. I'll probably give this a try at the weekend as I have a release for both of those two, and they are always a pita to release.