Publish an Eclipse p2 composite repository on Bintray

In a previous post I showed how to manage an Eclipse composite p2 repository and how to publish an Eclipse p2 composite repository on Sourceforge. In this post I’ll show a similar procedure to publish an Eclipse p2 composite repository on Bintray. The procedure is part of the Maven/Tycho build so that it is fully automated. Moreover, the pom.xml and the ant files can be fully reused in your own projects (just a few properties have to be adapted).

The complete example at https://github.com/LorenzoBettini/p2composite-bintray-example.

First of all, this procedure is quite different from the ones shown in other blogs (e.g., this one, this one and this one): in those approaches the p2 metadata (i.e., artifacts.jar and content.jar) are uploaded independently from a version, always in the same directory, thus overwriting the existing metadata. This leads to the fact that only the latest version of published features and bundles will be available to the end user. This is quite against the idea that old versions should still be available, and in general, all the versions should be available for the end users, especially if a new version has some breaking change and the user is not willing to update (see p2’s do’s and do not’s). For this reason, I always publish p2 composite repositories.

Quoting from https://wiki.eclipse.org/Equinox/p2/Composite_Repositories_(new)

The goal of composite repositories is to make this task easier by allowing you to have a parent repository which refers to multiple children. Users are then able to reference the parent repository and the children’s content will transparently be available to them.

In order to achieve this, all published p2 repositories must be available, each one with their own p2 metadata that should never be overwritten.

On the contrary, the metadata that we will overwrite will be the one for the composite metadata, i.e., compositeContent.xml and compositeArtifacts.xml.

In this example, all the binary artifacts are meant to be made available from: https://dl.bintray.com/lorenzobettini/p2-composite-example/. (NOTE: the artifacts are not effectively available from that URL anymore).

Directory Structure

What I aim at is to have the following remote paths on Bintray:

  • releases: in this directory all p2 simple repositories will be uploaded, each one in its own directory, named after version.buildQualifier, e.g., 1.0.0.v20160129-1616/ etc. Your Eclipse users can then use the URL of one of these single update sites to stick to that specific version.
  • updates: in this directory the composite metadata will be uploaded. The URL https://dl.bintray.com/lorenzobettini/p2-composite-example/updates/ should be used by your Eclipse users to install the features in their Eclipse of for target platform resolution (depending on the kind of projects you’re developing). All versions will be available from this composite update site; I call this main composite. Moreover, you can provide the URL to a child composite update site that includes all versions for a given major.minor stream, e.g., https://dl.bintray.com/lorenzobettini/p2-composite-example/updates/1.0/, https://dl.bintray.com/lorenzobettini/p2-composite-example/updates/1.1/, etc. I call each one of these, child composite.
  • zipped: in this directory we will upload the zipped p2 repository for each version.

Summarizing we’ll end up with a remote directory structure like the following

Uploading using REST API

In the posts I mentioned above, the typical line to upload contents with the REST API is of the shape

For metadata, and

For features and plugins.

But this has the drawback I was mentioning above.

Thanks to the Bintray Support, I managed to use a different scheme that allows me to store p2 metadata for a single p2 repository in the same directory of the p2 repository itself and to keep those metadata separate for each single release.

To achieve this, we need to use another URL scheme for uploading, using matrix params options or header options.

This means that we’ll upload everything with this URL

On the contrary, for uploading p2 composite metadata, we’ll use the schema of the other approaches, i.e., we will not associate it to any specific version; we just need to specify the desired remote path where we’ll upload the main and the child composite metadata.

Building Steps

During the build, we’ll have to update the composite site metadata, and we’ll have to do that locally.

The steps that we’ll perform during the Maven/Tycho build, which will rely on some Ant scripts can be summarized as follows:

  • Retrieve the remote composite metadata compositeContent/Artifacts.xml, both for the main composite and the child composite. If these metadata cannot be found remotely, we fail gracefully: it means that it is the first time we release, or, if only the child composite cannot be found, that we’re releasing a new major.minor version. These will be downloaded in the directories target/main-composite and target/child-composite respectively. These will be created anyway.
  • Preprocess possible downloaded composite metadata: if this property is present

    We must temporarily set it to false, otherwise we will not be able to add additional elements in the composite site with the p2 ant tasks.
  • Update the composite metadata using the version information passed from the Maven/Tycho build using the p2 Ant tasks for composite repositories
  • Post process the composite metadata (i.e., put the property p2.atomic.composite.loading above to true, see https://bugs.eclipse.org/bugs/show_bug.cgi?id=356561 for further details about this property). UPDATE: Please have a look at the comment section, in particular, the comments from pascalrapicault, about this property.
  • Upload everything to bintray: both the new p2 repository, its zipped version and all the composite metadata.

IMPORTANT: the pre and post processing of composite metadata that we’ll implement assumes that such metadata are not compressed. Anyway, I always prefer not to compress the composite metadata since it’s easier, later, to manually change them or reviewing.

Technical Details

You can find the complete example at https://github.com/LorenzoBettini/p2composite-bintray-example. Here I’ll sketch the main parts. First of all, all the mechanisms for updating the composite metadata and pushing to Bintray (i.e., the steps detailed above) are in the project p2composite.example.site, which is a Maven/Tycho project with eclipse-repository packaging.

The pom.xml has some properties that you should adapt to your project, and some other properties that can be left as they are if you’re OK with the defaults:

If you change the default remote paths it is crucial that you update the child.repository.path.prefix consistently. In fact, this is used to update the composite metadata for the composite children. For example, with the default properties the composite metadata will look like the following (here we show only compositeContent.xml):

You can also see that two crucial properties, bintray.user and, in particular, bintray.apikey should not be made public. You should keep these hidden, for example, you can put them in your local .m2/settings.xml file, associated to the Maven profile that you use for releasing (as illustrated in the following). This is an example of settings.xml

In the pom.xml of this project there is a Maven profile, release-composite, that should be activated when you want to perform the release steps described above.

We also make sure that the generated zipped p2 repository has a name with fully qualified version

In the release-composite Maven profile, we use the maven-antrun-plugin to execute some ant targets (note that the Maven properties are automatically passed to the Ant tasks): one to retrieve the remote composite metadata, if they exist, and the other one as the final step to deploy the p2 repository, its zipped version and the composite metadata to Bintray:

The Ant tasks are defined in the file bintray.ant. Please refer to the example for the complete file. Here we sketch the main parts.

This Ant file relies on some properties with default values, and other properties that are expected to be passed when running these tasks, i.e., from the pom.xml

To retrieve the existing remote composite metadata we execute the following, using the standard Ant get task. Note that if there is no composite metadata (e.g., it’s the first release that we execute, or we are releasing a new major.minor version so there’s no child composite for that version) we ignore the error; however, we still create the local directories for the composite metadata:

For preprocessing/postprocessing composite metadata (in order to deal with the property p2.atomic.composite.loading as explained in the previous section) we have

Finally, to push everything to Bintray we execute curl with appropriate URLs, as we described in the previous section about REST API. The single tasks for pushing to Bintray are similar, so we only show one for uploading the p2 repository associated to a specific version, and the one for uploading p2 composite metadata. As detailed at the beginning of the post, we use different URL shapes.

To update composite metadata we execute an ant task using the tycho-eclipserun-plugin. This way, we can execute the Eclipse application org.eclipse.ant.core.antRunner, so that we can execute the p2 Ant tasks for managing composite repositories.

ATTENTION: in the following snipped, for the sake of readability, I split the <appArgLine> into several lines, but in your pom.xml it must be exactly in one (long) line.

The file packaging-p2-composite.ant is similar to the one I showed in a previous post. We use the p2 Ant tasks for adding a child to a composite p2 repository (recall that if there is no existing composite repository, the task for adding a child also creates new compositeContent.xml/Artifacts.xml; if a child with the same name exists the ant task will not add anything new).

Removing Released artifacts

In case you want to remove an existing released version, since we upload the p2 repository and the zipped version as part of a package’s version, we just need to delete that version using the Bintray Web UI. However, this procedure will never remove the metadata, i.e., artifacts.jar and content.jar. The same holds if you want to remove the composite metadata. For these metadata files, you need to use the REST API, e.g., with curl. I put a shell script in the example to quickly remove all the metadata files from a given remote Bintray directory.

Performing a Release

For performing a release you just need to run

on the p2composite.example.tycho project.

Concluding Remarks

As I said, the procedure shown in this example is meant to be easily reusable in your projects. The ant files can be simply copied as they are. The same holds for the Maven profile. You only need to specify the Maven properties that contain values for your very project, and adjust your settings.xml with sensitive data like the bintray APIKEY.

Happy Releasing! 🙂

16 thoughts on “Publish an Eclipse p2 composite repository on Bintray

  1. pascalrapicault

    I would recommend removing the or always setting it to false since the composites you are pushing are standalone, and this is the reality of most cases. This will make the composite more resilient to failures.

    Reply
    1. Lorenzo Bettini Post author

      Hi Pascal,
      probably the WordPress editor removed xml tags… did you mean to always set p2.atomic.composite.loading to false? I seemed to understand that properties should always be set to true, and indeed, it seems to be the default when you create a composite update site with the p2 ant tasks.
      If I understand that property correctly, if I pushed two versions, say 1.1 and 1.2, and during p2 resolution 1.2 becomes unavailable (e.g., due to a network error), this property set to true will make the resolution fail. Without this property (or set it to false) then p2 will not take the latest version. This way the builds using my update site won’t be always reproducible.
      Or did I get the meaning of that property wrong?
      cheers
      Lorenzo

      Reply
      1. pascalrapicault

        I was indeed talking about the atomic.composite.loading property. The point of the property is to not to act on dependency resolution but on how the repository should be loaded. If we take the case of a simple repo (not composite), if the content.jar fails to load, the repository will fail to load, the user will have a message saying the repo can’t be reached, end of story.

        Now for composite repos, there were cases where people would be composing repos out of multiple repositories and wanted the same behaviour than for a simple repo in terms of loading consistency. For example people wanted to have a composite repo with one child repo for core stuffs, another one for UI, etc. and in this case it was paramount that all the content.jar of each simple repo was loaded successfully before doing anything, otherwise things would fail to resolve. This is why this flag was added.

        Now for the case of publishing multiple builds of the same software, my argument is that it is not necessary to have this flag because even if one build fails to load, it is not the end of the world. Say build 20 is failing to load but I’m at build 25, it does not worth breaking the build for that.

        I understand the point on build reproducibility but this property is not enough to support this. This is simply because the composite repo could suddenly have a new child like build 26 and the consumer could also get surprised by this. IMO, someone really caring about reproducible build should either use a specific composite repo and/or (for the paranoid, use the and) lock the version of the consumed dependency in the target. Actually the real paranoid will have a repo only containing what they want and that they control 🙂

        Reply
    1. Lorenzo Bettini Post author

      I’m glad you liked that, especially because it is appreciated by a p2 committer!
      Thank you for p2! 🙂

      Reply
  2. Konrad Windsyus

    Thanks a lot for your article.
    Unfortunately I get the following issue when trying to upload during the Ant task “push-p2-repo-to-bintray”.

    … {“message”:”Failed to resolve package name”} …

    This is the first time I am trying a release, and in bintray I only created a generic repository manually beforehand. Do I need to create the package as well manually? I found a similar issue in https://github.com/joshmarshall/bintray-upload/issues/1 which points in that direction.
    Any ideas?

    P.S. It would be great if you could extend your article with how exactly the repository must be created (which type it should be).

    Reply
    1. Konrad Windszus

      I figured out that the repository and the package (which must be named “releases” as well) must be initially created manually. Maybe you can add a hint to this blog article. Thanks.

      Reply
  3. kthoms

    Hi Lorenzo! Thanks again for sharing this article. I was able to use it for https://github.com/itemis/xtext-testing
    One remark: I prefer to bind the “deploy-repository” task process to the “deploy” phase instead of “verify”. It is not expected that “verify” does an upload. However, doing this requires to disable the maven-deploy-plugin. This should be done in pluginManagement in the parent POM by setting the skip flag (see http://maven.apache.org/plugins/maven-deploy-plugin/faq.html)

    ~Karsten

    Reply
    1. Lorenzo Bettini Post author

      Glad you enjoyed that Karsten and that you were able to reuse it (seamlessly, I hope)! Indeed, it took me quite some time to understand how to do that at the beginning 😉
      You’re right about the deploy phase, but I usually avoid that because that would trigger the deployment of Maven artifacts (I usually also avoid install since I don’t want my artifacts to go into the local cache repository), or it would require to explicitly skip, that as you suggest. Please, feel free to contribute to the example with a PR, especially concerning the skip of deploy phase!

      Reply
  4. Nicolas Rouquette

    FYI: I found it helpful to add some logging to the curl commands.

    Right now, you have:

    I use instead:

    But since this can produce too much useless output, I filter this using awk:

    awk ‘
    BEGIN {flag=0}
    /\[apply\]\ (\*|\{|\})/{next}
    /\[apply\]\ > (Host|Authorization|User-Agent|Accept|Content-Length|Expect):/{next}
    /\[apply\]\ 0)print}
    {if(flag==0)print}’

    Reply
    1. Lorenzo Bettini

      Nicholas, unfortunately WordPress removed most of your code snippets… could you please post your code on pastebin or some github gist?

      Reply
  5. Pingback: Bintray支持Eclipse p2存储库 - 算法网

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.