Tag Archives: p2

Eclipse p2 site references

Say you publish a p2 repository for your Eclipse bundles and features. Typically your bundles and features will depend on something external (other Eclipse bundles and features). The users of your p2 repository will have to also use the p2 repositories of the dependencies of your software otherwise they won’t be able to install your software. If your software only relies on standard Eclipse bundles and features, that is, something that can be found in the standard Eclipse central update site, you should have no problem: your users will typically have the Eclipse central update site already configured in their Eclipse installations. So, unless your software requires a specific version of an Eclipse dependency, you should be fine.

What happens instead if your software relies on external dependencies that are available only in other p2 sites? Or, put it another way, you rely on an Eclipse project that is not part of the simultaneous release or you need a version different from the one provided by a specific Eclipse release.

You should tell your users to use those specific p2 sites as well. This, however, will decrease the user experience at least from the installation point of view. One would like to use a p2 site and install from it without further configurations.

To overcome this issue, you should make your p2 repository somehow self-contained. I can think of 3 alternative ways to do that:

  • If you build with Tycho (which is probably the case if you don’t do releng stuff manually), you could use <includeAllDependencies> of the tycho-p2-repository plugin to “to aggregate all transitive dependencies, making the resulting p2 repository self-contained.” Please keep in mind that your p2 repository itself will become pretty huge (likely a few hundred MB), so this might not be feasible in every situation.
  • You can put the required p2 repositories as children of your composite update site. This might require some more work and will force you to introduce composite update sites just for this. I’ve written about p2 composite update sites many times in this blog in the past, so I will not consider this solution further.
  • You can use p2 site references that are meant just for the task mentioned so far and that have been introduced in the category.xml specification for some time now. The idea is that you put references to the p2 sites of your software dependencies and the corresponding content metadata of the generated p2 repository will contain links to the p2 sites of dependencies. Then, p2 will automatically contact those sites when installing software (at least from Eclipse, from the command line we’ll have to use specific arguments as we’ll see later). Please keep in mind that this mechanism works only if you use recent versions of Eclipse (if I remember correctly this has been added a couple of years ago).

In this blog post, I’ll describe such a mechanism, in particular, how this can be employed during the Tycho build.

The simple project used in this blog post can be found here: https://github.com/LorenzoBettini/tycho-site-references-example. You should be able to easily reuse most of the POM stuff in your own projects.

IMPORTANT: To benefit from this, you’ll have to use at least Tycho 2.4.0. In fact, Tycho started to support site references only a few versions ago, but only in version 2.4.0 this has been implemented correctly. (I personally fixed this: https://github.com/eclipse/tycho/issues/141.) If you use a (not so) older version, e.g., 2.3.0, there’s a branch in the above GitHub repository, tycho-2.3.0, where some additional hacks have to be performed to make it work (rewrite metadata contents and re-compress the XML files, just to mention a few), but I’d suggest you use Tycho 2.4.0.

There’s also another important aspect to consider: if your software switches to a different version of a dependency that is available on a different p2 repository, you have to update such information consistently. In this blog post, we’ll deal with this issue as well, keeping it as automatic (i.e., less error-prone) as possible.

The example project

The example project is very simple:

  • parent project with the parent POM;
  • a plugin project created with the Eclipse wizard with a simple handler (so it depends on org.eclipse.ui and org.eclipse.core.runtime);
  • a feature project including the plugin project. To make the example more interesting this feature also requires, i.e., NOT includes, the external feature org.eclipse.xtext.xbase. We don’t actually use such an Xtext feature, but it’s useful to recreate an example where we need a specific p2 site containing that feature;
  • a site project with category.xml that is used to generate during the Tycho build our p2 repository.

To make the example interesting the dependency on the Xbase feature is as follows

So we require version 2.25.0.

The target platform is defined directly in the parent POM as follows (again, to keep things simple):

Note that I explicitly added the Xtext 2.25.0 site repository because in the 2020-12 Eclipse site Xtext is available with a lower version 2.24.0.

This defines the target platform we built (and in a real example, hopefully, tested) our bundle and feature.

The category.xml initially is defined as follows

The problem

If you generate the p2 repository with the Maven/Tycho build, you will not be able to install the example feature unless Xtext 2.25.0 and its dependencies can be found (actually, also the standard Eclipse dependencies have to be found, but as said above, the Eclipse update site is already part of the Eclipse distributions). You then need to tell your users to first add the Xtext 2.25.0 update site. In the following, we’ll handle this.

A manual, and thus cumbersome, way to verify that is to try to install the example feature in an Eclipse installation pointing to the p2 repository generated during the build. Of course, we’ll keep also this verification mechanism automatic and easy. So, before going on, following a Test-Driven approach (which I always love), let’s first reproduce the problem in the Tycho build, by adding this configuration to the site project (plug-in versions are configured in the pluginManagement section of the parent POM):

The idea is to run the standard Eclipse p2 director application through the tycho-eclipserun-plugin. The dependency configuration is standard for running such an Eclipse application. We try to install our example feature from our p2 repository into a temporary output directory (these values are defined as properties so that you can copy this plugin configuration in your projects and simply adjust the values of the properties). Also, the arguments passed to the p2 director are standard and should be easy to understand. The only non-standard argument is -followReferences that will be crucial later (for this first run it would not be needed).

Running mvn clean verify should now highlight the problem:

This would mimic the situation your users might experience.

The solution

Let’s fix this: we add to the category.xml the references to the same p2 repositories we used in our target platform. We can do that manually (or by using the Eclipse Category editor, in the tab Repository Properties):

The category.xml initially is defined as follows

Now when we create the p2 repository during the Tycho build, the content.xml metadata file will contain the references to the p2 repository (with a syntax slightly different, but that’s not important; it will contain a reference to the metadata repository and to the artifact repository, which usually are the same). Now, our users can simply use our p2 repository without worrying about dependencies! Our p2 repository will be self-contained.

Let’s verify that by running mvn clean verify; now everything is fine:

Note that this requires much more time: now the p2 director has to contact all the p2 sites defined as references and has to also download the requirements during the installation. We’ll see how to optimize this part as well.

In the corresponding output directory, you can find the installed plugins; you can’t do much with such installed bundles, but that’s not important. We just want to verify that our users can install our feature simply by using our p2 repository, that’s all!

You might not want to run this verification on every build, but, for instance, only during the build where you deploy the p2 repository to some remote directory (of course, before the actual deployment step). You can easily do that by appropriately configuring your POM(s).

Some optimizations

As we saw above, each time we run the clean build, the verification step has to access remote sites and has to download all the dependencies. Even though this is a very simple example, the dependencies during the installation are almost 100MB. Every time you run the verification. (It might be the right moment to stress that the p2 director will know nothing about the Maven/Tycho cache.)

We can employ some caching mechanisms by using the standard mechanism of p2: bundle pool! This way, dependencies will have to be downloaded only the very first time, and then the cached versions will be used.

We simply introduce another property for the bundle pool directory (I’m using by default a hidden directory in the home folder) and the corresponding argument for the p2 director application:

Note that now the plug-ins during the verification step will NOT be installed in the specified output directory (which will store only some p2 properties and caches): they will be installed in the bundle pool directory. Again, as said above, you don’t need to interact with such installed plug-ins, you only need to make sure that they can be installed.

In a CI server, you should cache the bundle pool directory as well if you want to benefit from some speed. E.g., this example comes with a GitHub Actions workflow that stores also the bundle pool in the cache, besides the .m2 directory.

This will also allow you to easily experiment with different configurations of the site references in your p2 repository. For example, up to now, we put the same sites used for the target platform. Referring to the whole Eclipse releases p2 site might be too much since it contains all the features and bundles of all the projects participating in Eclipse Simrel. In the target platform, this might be OK since we might want to use some dependencies only for testing. For our p2 repository, we could tweak references so that they refer only to the minimal sites containing all our features’ requirements.

For this example we can replace the 2 sites with 4 small sites with all the requirements (actually the Xtext 2.25.0 is just the same as before):

You can verify that removing any of them will lead to installation failures.

The first time this tweaking might require some time, but you now have an easy way to test this!

Keeping things consistent

When you update your target platform, i.e., your dependencies versions, you must make sure to update the site references in the category.xml accordingly. It would be instead nice to modify this information in a single place so that everything else is kept consistent!

We can use again properties in the parent POM:

We want to rely on such properties also in the category.xml, relying on the Maven standard mechanism of copy resources with filtering.

We create another category.xml in the subdirectory templates of the site project using the above properties in the site references (at least in the ones where we want to have control on a specific version):

and in the site project we configure the Maven resources plugin appropriately:

Of course, we execute that in a phase that comes BEFORE the phase when the p2 repository is generated. This will overwrite the standard category.xml file (in the root of the site project) by replacing properties with the corresponding values!

By the way, you could use the property eclipse-version also in the configuration of the Tycho Eclipserun plugin seen above, instead of hardcoding 2020-12.

Happy releasing! 🙂

Publishing an Eclipse p2 composite repository on GitHub Pages

I had already described the process of publishing an Eclipse p2 composite update site:

Well, now that Bintray is shutting down, and Sourceforge is quite slow in serving an Eclipse update site, I decided to publish my Eclipse p2 composite update sites on GitHub Pages.

GitHub Pages might not be ideal for serving binaries, and it has a few limitations. However, such limitations (e.g., published sites may be no larger than 1 GB, sites have a soft bandwidth limit of 100GB per month and sites have a soft limit of 10 builds per hour) are not that crucial for an Eclipse update site, whose artifacts are not that huge. Moreover, at least my projects are not going to serve more than 100GB per month, unfortunately, I might say 😉

In this tutorial, I’ll show how to do that, so that you can easily apply this procedure also to your projects!

The procedure is part of the Maven/Tycho build so that it is fully automated. Moreover, the pom.xml and the ant files can be fully reused in your own projects (just a few properties have to be adapted). The idea is that you can run this Maven build (basically, “mvn deploy”) on any CI server (as long as you have write-access to the GitHub repository hosting the update site – more on that later). Thus, you will not depend on the pipeline syntax of a specific CI server (Travis, GitHub Actions, Jenkins, etc.), though, depending on the specific CI server you might have to adjust a few minimal things.

These are the main points:

The p2 children repositories and the p2 composite repositories will be published with standard Git operations since we publish them in a GitHub repository.

Let’s recap what p2 composite update sites are. Quoting from https://wiki.eclipse.org/Equinox/p2/Composite_Repositories_(new)

As repositories continually grow in size they become harder to manage. The goal of composite repositories is to make this task easier by allowing you to have a parent repository which refers to multiple children. Users are then able to reference the parent repository and the children’s content will transparently be available to them.

In order to achieve this, all published p2 repositories must be available, each one with its own p2 metadata that should never be overwritten. On the contrary, the metadata that we will overwrite will be the one for the composite metadata, i.e., compositeContent.xml and compositeArtifacts.xml.

Directory Structure

I want to be able to serve these composite update sites:

  • the main one collects all the versions
  • a composite update site for each major version (e.g., 1.x, 2.x, etc.)
  • a composite update site for each major.minor version (e.g., 1.0.x, 1.1.x, 2.0.x, etc.)

What I aim at is to have the following paths:

  • releases: in this directory, all p2 simple repositories will be uploaded, each one in its own directory, named after version.buildQualifier, e.g., 1.0.0.v20210307-2037, 1.1.0.v20210307-2104, etc. Your Eclipse users can then use the URL of one of these single update sites to stick to that specific version.
  • updates: in this directory, the metadata for major and major.minor composite sites will be uploaded.
  • root: the main composite update site collecting all versions.

To summarize, we’ll end up with a remote directory structure like the following one

Thus, if you want, you can provide these sites to your users (I’m using the URLs that correspond to my example):

  • https://lorenzobettini.github.io/p2composite-github-pages-example-updates for the main global update site: every new version will be available when using this site;
  • https://lorenzobettini.github.io/p2composite-github-pages-example-updates/updates/1.x for all the releases with major version 1: for example, the user won’t see new releases with major version 2;
  • https://lorenzobettini.github.io/p2composite-github-pages-example-updates/updates/1.x/1.0.x for all the releases with major version 1 and minor version 0: the user will only see new releases of the shape 1.0.0, 1.0.1, 1.0.2, etc., but NOT 1.1.0, 1.2.3, 2.0.0, etc.

If you want to change this structure, you have to carefully tweak the ant file we’ll see in a minute.

Building Steps

During the build, before the actual deployment, we’ll have to update the composite site metadata, and we’ll have to do that locally.

The steps that we’ll perform during the Maven/Tycho build are:

  • Clone the repository hosting the composite update site (in this example, https://github.com/LorenzoBettini/p2composite-github-pages-example-updates);
  • Create the p2 repository (with Tycho, as usual);
  • Copy the p2 repository in the cloned repository in a subdirectory of the releases directory (the name of the subdirectory has the same qualified version of the project, e.g., 1.0.0.v20210307-2037);
  • Update the composite update sites information in the cloned repository (using the p2 tools);
  • Commit and push the updated clone to the remote GitHub repository (the one hosting the composite update site).

First of all, in the parent POM, we define the following properties, which of course you need to tweak for your own projects:

It should be clear which properties you need to modify for your project. In particular, the github-update-repo is the URL (with authentication information) of the GitHub repository hosting the composite update site, and the site.label is the label that will be put in the composite metadata.

Then, in the parent POM, we configure in the pluginManagement section all the versions of the plugin we are going to use (see the sources of the example on GitHub).

The most interesting configuration is the one for the tycho-packaging-plugin, where we specify the format of the qualified version:

Moreover, we create a profile release-composite (which we’ll also use later in the POM of the site project), where we disable the standard Maven plugins for install and deploy. Since we are going to release our Eclipse p2 composite update site during the deploy phase, but we are not interested in installing and deploying the Maven artifacts, we skip the standard Maven plugins bound to those phases:

The interesting steps are in the site project, the one with <packaging>eclipse-repository</packaging>. Here we also define the profile release-composite and we use a few plugins to perform the steps involving the Git repository described above (remember that these configurations are inside the profile release-composite, of course in the build plugins section):

Let’s see these configurations in detail. In particular, it is important to understand how the goals of the plugins are bound to the phases of the default lifecycle; remember that on the phase package, Tycho will automatically create the p2 repository and it will do that before any other goals bound to the phase package in the above configurations:

  • with the build-helper-maven-plugin we parse the current version of the project, in particular, we set the properties holding the major and minor versions that we need later to create the composite metadata directory structure; its goal is automatically bound to one of the first phases (validate) of the lifecycle;
  • with the exec-maven-plugin we configure the execution of the Git commands:
    • we clone the Git repository of the update site (with –depth=1 we only get the latest commit in the history, the previous commits are not interesting for our task); this is done in the phase pre-package, that is before the p2 repository is created by Tycho; the Git repository is cloned in the output directory target/checkout
    • in the phase verify (that is, after the phase package), we commit the changes (which will be done during the phase package as shown in the following points)
    • in the phase deploy (that is, the last phase that we’ll run on the command line), we push the changes to the Git repository of the update site
  • with the maven-resources-plugin we copy the p2 repository generated by Tycho into the target/checkout/releases directory in a subdirectory with the name of the qualified version of the project (e.g., 1.0.0.v20210307-2037);
  • with the tycho-eclipserun-plugin we create the composite metadata; we rely on the Eclipse application org.eclipse.ant.core.antRunner, so that we can execute the p2 Ant task for managing composite repositories (p2.composite.repository). The Ant tasks are defined in the Ant file packaging-p2composite.ant, stored in the site project. In this file, there are also a few properties that describe the layout of the directories described before. Note that we need to pass a few properties, including the site.label, the directory of the local Git clone, and the major and minor versions that we computed before.

Keep in mind that in all the above steps, non-existing directories will be automatically created on-demand (e.g., by the maven-resources-plugin and by the p2 Ant tasks). This means that the described process will work seamlessly the very first time when we start with an empty Git repository.

Now, from the parent POM on your computer, it’s enough to run

and the release will be performed. When cloning you’ll be asked for the password of the GitHub repository, and, if not using an SSH agent or a keyring, also when pushing. Again, this depends on the URL of the GitHub repository; you might use an HTTPS URL that relies on the GitHub token, for example.

If you want to make a few local tests before actually releasing, you might stop at the phase verify and inspect the target/checkout to see whether the directories and the composite metadata are as expected.

You might also want to add another execution to the tycho-eclipserun-plugin to add a reference to another Eclipse update site that is required to install your software. The Ant file provides a task for that, p2.composite.add.external that will store the reference into the innermost composite child (e.g., into 1.2.x); here’s an example that adds a reference to the Eclipse main update site:

For example, in my Xtext projects, I use this technique to add a reference to the Xtext update site corresponding to the Xtext version I’m using in that specific release of my project. This way, my update site will be “self-contained” for my users: when using my update site for installing my software, p2 will be automatically able to install also the required Xtext bundles!

Releasing from GitHub Actions

The Maven command shown above can be used to perform a release from your computer. If you want to release your Eclipse update site directly from GitHub Actions, there are a few more things to do.

First of all, we are talking about a GitHub Actions workflow stored and executed in the GitHub repository of your project, NOT in the GitHub repository of the update site. In this example, it is https://github.com/LorenzoBettini/p2composite-github-pages-example.

In such a workflow, we need to push to another GitHub repository. To do that

  • create a GitHub personal access token (selecting repo);
  • create a secret in the GitHub repository of the project (where we run the GitHub Actions workflow), in this example it is called ACTIONS_TOKEN, with the value of that token;
  • when running the Maven deploy command, we need to override the property github-update-repo by specifying a URL for the GitHub repository with the update site using the HTTPS syntax and the encrypted ACTIONS_TOKEN; in this example, it is https://x-access-token:${{ secrets.ACTIONS_TOKEN }}@github.com/LorenzoBettini/p2composite-github-pages-example-updates;
  • we also need to configure in advance the Git user and email, with some values, otherwise, Git will complain when creating the commit.

To summarize, these are the interesting parts of the release.yml workflow (see the full version here: https://github.com/LorenzoBettini/p2composite-github-pages-example/blob/master/.github/workflows/release.yml):

The workflow is configured to be executed only when you push to the release branch.

Remember that we are talking about the Git repository hosting your project, not the one hosting your update site.

Final thoughts

With the procedure described in this post, you publish your update sites and the composite metadata during the Maven build, so you never deal manually with the GitHub repository of your update site. However, you can always do that! For example, you might want to remove a release. It’s just a matter of cloning that repository, do your changes (i.e., remove a subdirectory of releases and update manually the composite metadata accordingly), commit, and push. Now and then you might also clean up the history of such a Git repository (the history is not important in this context), by pushing with –force after resetting the Git history. By the way, by tweaking the configurations above you could also do that every time you do a release: just commit with amend and push force!

Finally, you could also create an additional GitHub repository for snapshot releases of your update sites, or for milestones, or release candidate.

Happy releasing! 🙂

Publish an Eclipse p2 composite repository on Bintray

In a previous post I showed how to manage an Eclipse composite p2 repository and how to publish an Eclipse p2 composite repository on Sourceforge. In this post I’ll show a similar procedure to publish an Eclipse p2 composite repository on Bintray. The procedure is part of the Maven/Tycho build so that it is fully automated. Moreover, the pom.xml and the ant files can be fully reused in your own projects (just a few properties have to be adapted).

The complete example at https://github.com/LorenzoBettini/p2composite-bintray-example.

First of all, this procedure is quite different from the ones shown in other blogs (e.g., this one, this one and this one): in those approaches the p2 metadata (i.e., artifacts.jar and content.jar) are uploaded independently from a version, always in the same directory, thus overwriting the existing metadata. This leads to the fact that only the latest version of published features and bundles will be available to the end user. This is quite against the idea that old versions should still be available, and in general, all the versions should be available for the end users, especially if a new version has some breaking change and the user is not willing to update (see p2’s do’s and do not’s). For this reason, I always publish p2 composite repositories.

Quoting from https://wiki.eclipse.org/Equinox/p2/Composite_Repositories_(new)

The goal of composite repositories is to make this task easier by allowing you to have a parent repository which refers to multiple children. Users are then able to reference the parent repository and the children’s content will transparently be available to them.

In order to achieve this, all published p2 repositories must be available, each one with their own p2 metadata that should never be overwritten.

On the contrary, the metadata that we will overwrite will be the one for the composite metadata, i.e., compositeContent.xml and compositeArtifacts.xml.

In this example, all the binary artifacts are meant to be made available from: https://dl.bintray.com/lorenzobettini/p2-composite-example/. (NOTE: the artifacts are not effectively available from that URL anymore).

Directory Structure

What I aim at is to have the following remote paths on Bintray:

  • releases: in this directory all p2 simple repositories will be uploaded, each one in its own directory, named after version.buildQualifier, e.g., 1.0.0.v20160129-1616/ etc. Your Eclipse users can then use the URL of one of these single update sites to stick to that specific version.
  • updates: in this directory the composite metadata will be uploaded. The URL https://dl.bintray.com/lorenzobettini/p2-composite-example/updates/ should be used by your Eclipse users to install the features in their Eclipse of for target platform resolution (depending on the kind of projects you’re developing). All versions will be available from this composite update site; I call this main composite. Moreover, you can provide the URL to a child composite update site that includes all versions for a given major.minor stream, e.g., https://dl.bintray.com/lorenzobettini/p2-composite-example/updates/1.0/, https://dl.bintray.com/lorenzobettini/p2-composite-example/updates/1.1/, etc. I call each one of these, child composite.
  • zipped: in this directory we will upload the zipped p2 repository for each version.

Summarizing we’ll end up with a remote directory structure like the following

Uploading using REST API

In the posts I mentioned above, the typical line to upload contents with the REST API is of the shape

For metadata, and

For features and plugins.

But this has the drawback I was mentioning above.

Thanks to the Bintray Support, I managed to use a different scheme that allows me to store p2 metadata for a single p2 repository in the same directory of the p2 repository itself and to keep those metadata separate for each single release.

To achieve this, we need to use another URL scheme for uploading, using matrix params options or header options.

This means that we’ll upload everything with this URL

On the contrary, for uploading p2 composite metadata, we’ll use the schema of the other approaches, i.e., we will not associate it to any specific version; we just need to specify the desired remote path where we’ll upload the main and the child composite metadata.

Building Steps

During the build, we’ll have to update the composite site metadata, and we’ll have to do that locally.

The steps that we’ll perform during the Maven/Tycho build, which will rely on some Ant scripts can be summarized as follows:

  • Retrieve the remote composite metadata compositeContent/Artifacts.xml, both for the main composite and the child composite. If these metadata cannot be found remotely, we fail gracefully: it means that it is the first time we release, or, if only the child composite cannot be found, that we’re releasing a new major.minor version. These will be downloaded in the directories target/main-composite and target/child-composite respectively. These will be created anyway.
  • Preprocess possible downloaded composite metadata: if this property is present

    We must temporarily set it to false, otherwise we will not be able to add additional elements in the composite site with the p2 ant tasks.
  • Update the composite metadata using the version information passed from the Maven/Tycho build using the p2 Ant tasks for composite repositories
  • Post process the composite metadata (i.e., put the property p2.atomic.composite.loading above to true, see https://bugs.eclipse.org/bugs/show_bug.cgi?id=356561 for further details about this property). UPDATE: Please have a look at the comment section, in particular, the comments from pascalrapicault, about this property.
  • Upload everything to bintray: both the new p2 repository, its zipped version and all the composite metadata.

IMPORTANT: the pre and post processing of composite metadata that we’ll implement assumes that such metadata are not compressed. Anyway, I always prefer not to compress the composite metadata since it’s easier, later, to manually change them or reviewing.

Technical Details

You can find the complete example at https://github.com/LorenzoBettini/p2composite-bintray-example. Here I’ll sketch the main parts. First of all, all the mechanisms for updating the composite metadata and pushing to Bintray (i.e., the steps detailed above) are in the project p2composite.example.site, which is a Maven/Tycho project with eclipse-repository packaging.

The pom.xml has some properties that you should adapt to your project, and some other properties that can be left as they are if you’re OK with the defaults:

If you change the default remote paths it is crucial that you update the child.repository.path.prefix consistently. In fact, this is used to update the composite metadata for the composite children. For example, with the default properties the composite metadata will look like the following (here we show only compositeContent.xml):

You can also see that two crucial properties, bintray.user and, in particular, bintray.apikey should not be made public. You should keep these hidden, for example, you can put them in your local .m2/settings.xml file, associated to the Maven profile that you use for releasing (as illustrated in the following). This is an example of settings.xml

In the pom.xml of this project there is a Maven profile, release-composite, that should be activated when you want to perform the release steps described above.

We also make sure that the generated zipped p2 repository has a name with fully qualified version

In the release-composite Maven profile, we use the maven-antrun-plugin to execute some ant targets (note that the Maven properties are automatically passed to the Ant tasks): one to retrieve the remote composite metadata, if they exist, and the other one as the final step to deploy the p2 repository, its zipped version and the composite metadata to Bintray:

The Ant tasks are defined in the file bintray.ant. Please refer to the example for the complete file. Here we sketch the main parts.

This Ant file relies on some properties with default values, and other properties that are expected to be passed when running these tasks, i.e., from the pom.xml

To retrieve the existing remote composite metadata we execute the following, using the standard Ant get task. Note that if there is no composite metadata (e.g., it’s the first release that we execute, or we are releasing a new major.minor version so there’s no child composite for that version) we ignore the error; however, we still create the local directories for the composite metadata:

For preprocessing/postprocessing composite metadata (in order to deal with the property p2.atomic.composite.loading as explained in the previous section) we have

Finally, to push everything to Bintray we execute curl with appropriate URLs, as we described in the previous section about REST API. The single tasks for pushing to Bintray are similar, so we only show one for uploading the p2 repository associated to a specific version, and the one for uploading p2 composite metadata. As detailed at the beginning of the post, we use different URL shapes.

To update composite metadata we execute an ant task using the tycho-eclipserun-plugin. This way, we can execute the Eclipse application org.eclipse.ant.core.antRunner, so that we can execute the p2 Ant tasks for managing composite repositories.

ATTENTION: in the following snipped, for the sake of readability, I split the <appArgLine> into several lines, but in your pom.xml it must be exactly in one (long) line.

The file packaging-p2-composite.ant is similar to the one I showed in a previous post. We use the p2 Ant tasks for adding a child to a composite p2 repository (recall that if there is no existing composite repository, the task for adding a child also creates new compositeContent.xml/Artifacts.xml; if a child with the same name exists the ant task will not add anything new).

Removing Released artifacts

In case you want to remove an existing released version, since we upload the p2 repository and the zipped version as part of a package’s version, we just need to delete that version using the Bintray Web UI. However, this procedure will never remove the metadata, i.e., artifacts.jar and content.jar. The same holds if you want to remove the composite metadata. For these metadata files, you need to use the REST API, e.g., with curl. I put a shell script in the example to quickly remove all the metadata files from a given remote Bintray directory.

Performing a Release

For performing a release you just need to run

on the p2composite.example.tycho project.

Concluding Remarks

As I said, the procedure shown in this example is meant to be easily reusable in your projects. The ant files can be simply copied as they are. The same holds for the Maven profile. You only need to specify the Maven properties that contain values for your very project, and adjust your settings.xml with sensitive data like the bintray APIKEY.

Happy Releasing! 🙂

Using the new Eclipse Installer

I’ve just started using the brand new Eclipse installer, and I’d like to report my experiences here. First of all, a big praise to Ed Merks and Eike Stepper for creating Oomph, on which the installer is based. 🙂

First of all, the installer is currently available in the “Developer Builds” section:

eclipse-installer1

Once you downloaded it and extracted it, just run the executable oomph:

eclipse-installer2If you see an exclamation mark (on the top right corner), click on it, you’ll see some updates are available, so update it right away, and when it’s done, press OK to restart it.

The very same top right corner, also opens a menu for customization of some features, the one I prefer is the Bundle Pool, a cool feature that has been in Eclipse for so many years, and so very badly advertised, I’m afraid!

“p2 natively supports the notion of bundle pooling. When using bundle pooling, multiple applications share a common plugins directory where their software is stored. There is no duplication of content, and no duplicated downloads when upgrading software.”

One of the cool things of Oomph is that it natively and automatically supports bundle pools, it makes it really easy to manage them and makes installation faster and with less space requirements (what’s already been downloaded and installed won’t have to be downloaded and installed again for further Eclipse installations).

eclipse-installer3

If you select that menu item, you can manage your bundle pools; the installer already detected existing bundle pools (I’ve been using them myself, manually, for some time now, and it detected that):

eclipse-installer4

For this blog post I will create another bundle pool, just for testing. To create a new bundle pool, you first need to create a new p2 agent; the agent is responsible to manage the bundle pool, and to keep track of all the bundles that a specific Eclipse installation requires (this is also known as a p2 profile).

So I select “New Agent…” and choose a location in my hard disk; this will also set a bundle pool:

eclipse-installer5

Just for demonstration, I’ll select the “pool”, “Delete…”, and create a “New Bundle Pool…” for the new agent, in another directory:

eclipse-installer6

Then I select the new bundle pool, and press “OK”.

From now on, all the installations will be managed by the new agent, and all bundles will be stored in the new bundle pool.

OK, now, back to the main window, let’s start installing “Eclipse IDE for Java Developers”

In the next windows, I choose to install the new Eclipse in a different folder from the proposed default:

eclipse-installer7

Let’s press “INSTALL”, and accept the LICENSE, the installation starts:

eclipse-installer8You’ll see that the installer is really quick (as far as I know, Oomph improved p2 internal mechanisms). It only took about a minute to install this Eclipse on my computer.

Then, you’re ready to launch this installation, or see the installation log.

eclipse-installer9But first, let’s have a look at the directory layout:

eclipse-installer10

you see that the installed eclipse does not have the typical directory structure: it has no “features”/”plugins” directories: these are in the shared bundle pool. Also note that the p2 agent location has a directory representing the profile of the installed Eclipse.

Let’s try and install another Eclipse, e.g., the “Eclipse DSL Tools” (what else if not the one with the cool Xtext framework? 😉

The dialog proposes an installation directory based on my previous choice; I also select “Luna” as the platform:

eclipse-installer11

Let’s press “INSTALL”… WOW! This time it’s even faster! You know why: only the new bundles are downloaded, everything else is shared. This also means: less space wasted on your hard disk! 🙂

But there are cooler things: Bundle pool management!

Go back to the “Bundle Pool Management” dialog, select the checkbox “Show Profiles” and you see the profiles handled by the current agent:

eclipse-installer12Select the agent and press “Analyze…”

You can see the bundles used by which profile:

eclipse-installer13Hope you enjoy this new installer! 🙂

 

 

 

 

Deploy your own custom Eclipse

This is the follow up of my previous post about building a custom Eclipse distribution. In this post I’ll show how to deploy the p2 site and the zipped products on Sourceforge. Concerning the p2 site, I’ll use the same technique, with some modifications, for building a composite update site and deploy it with rsync that I showed on another post.

In particular, we’ll accomplish several tasks:

  • creating and deploying the update site with only the features (without the products)
  • creating and deploying the update site including product definition and the zipped provisioned products
  • creating a self-contained update site (including all the dependencies)
  • providing an ant script for installing your custom Eclipse from the net

The code of the example can be found at: https://github.com/LorenzoBettini/customeclipse-example. In particular, I’ll start from where I left in the previous post.

The source code assumes a specific remote directory on Sourceforge, that is part of one of my Sourceforge projects, and it is writable only with my username and password. If you want to test this example, you can simply modify the property remote.dir in the parent pom specifying a local path in your computer (or by passing a value to the maven command with the syntax -Dremote.dir=<localpath>). Indeed, rsync can also synchronize two local directories.

Recall that when you perform a synchronization, specifying the wrong local directory might lead to a complete deletion of that directory. Moreover, source and destinations URLs in rsync have a different semantics depending on whether they terminate with a slash or not, so make sure you understand them if you need to customize this ant file or to pass special URLs.  

Creating and Deploying the p2 composite site

This part reuses most of what I showed in the previous posts:

In this blog post we want to be able to add a new p2 site to the composite update site (and deploy it) for two different projects:

  • customeclipse.example.site: This is the update site with only our features and bundles
  • customeclipse.example.ide.site: This is the update site with our features and bundles and the Eclipse product definition.

To reuse the ant files for managing the p2 composite update site and syncing it with rsync, and the Maven executions that use such ant files, we put the ant files in the parent project customeclipse.example.tycho, and we configure the Maven executions in the pluginManagement section of the parent pom.

We also put in the parent pom all the properties we’ll use for the p2 composite site and for rsync (again, please have a look at the previous posts for their meaning)

The pluginManagement section contains the configuration for managing the composite update site.

ATTENTION: in the following snipped, for the sake of readability, I split the <appArgLine> into several lines, but in your pom.xml it must be exactly in one (long) line.

The pluginManagement section also contains the configuration for updating and committing the composite update site to Sourceforge.

Now, we can simply activate such plugins in the build sections of our site projects described above.

In particular, we activate such plugins only inside profiles; for example, in the customeclipse.example.site project we have:

In customeclipse.example.ide.site we have similar sections, but the profiles are called differently, release-ide-composite and deploy-ide-composite, respectively.

So, if you want to update the p2 composite site with a new version containing only the features/bundles and deploy it on Sourceforge you need to run maven as follows

If you want to do the same, including the custom product definitions you need to run maven as follows (the additional build-ide profile is required because the customeclipse.example.ide.site is included as a Maven module only when that profile is activated; this way, products are created only when that profile is activated – just because provisioning a product requires some time and we don’t want to do that on normal builds)

NOTE: The remote directory on Sourceforge hosting  the composite update site will always be the same. This means that the local composite update site created and updated by both deploy-composite and deploy-ide-composite will be synchronized with the same remote folder.

In the customeclipse.example.ide.site, we added a p2.inf file with touchpoint instructions to add as update site in our Eclipse products the update site hosted on Sourceforge: http://sourceforge.net/projects/eclipseexamples/files/customeclipse/updates.

Deploying the zipped products

To copy the zipped products on Sourceforge we will still use rsync; actually, we won’t use any synchronization features: we only want to copy the zip files. I could have used the Ant Scp or Sftp tasks, but I experienced many problems with such tasks, so let’s use rsync also for that.

The ant file for rsync is slightly different with respect to the one shown in the previous post, since it has been refactored to pass the rsync macro more parameters. We still have the targets for update/commit synchronization; we added another target that will be used to simply copy something (i.e., the zipped products) to the remote directory, without any real synchronization. You may want to have a look at rsync documentation to fully understand the command line arguments.

In the customeclipse.example.ide.site, in the deploy-ide-composite profile, we configure another execution for the maven ant plugin (recall that in this profile the rsync synchronization configured in the parent’s pom pluginManagement section is also executed); this further execution will copy the zipped products to a remote folder on Sourceforge (as detailed in the previous post, you first need to create such folder using the Sourceforge web interface):

Note that when calling the rsync-copy-dir-contents of the rsync.ant file, we pass the properties as nested elements, in order to override their values (such properties’ value are already defined in the parent’s pom, and for this run we need to pass different values).

Now, if we run

many things will be executed:

  • rsync will synchronize our local composite update site with the remote composite update site
  • a new p2 site will be created, and added to our local composite update site
  • rsync will synchronize our local changes with the remote composite update site
  • Eclipse products will be created and zipped
  • the zipped products will be copied to Sourceforge

A self-contained p2 repository

Recall from the previous post that since in customeclipse.example.ide.feature we added Eclipse features (such as the platform and jdt) as dependencies (and not as included features), then the p2 update site we’ll create will not contain such features: it will contain only our own features and bundles. And that was actually intentional.

However, this means that the users of our features and of our custom Eclipse will still need to add the standard Eclipse update site before installing our features or updating the installed custom Eclipse.

If you want your p2 repository to be self-contained, i.e., to include also the external dependencies, you can do so by setting includeAllDependencies to true in the configuration of the tycho-p2-repository-plugin.

It makes sense to do that in the customeclipse.example.ide.site, so that all the dependencies for our custom Eclipse product will end up in the p2 repository:

However, doing so every time we add a new p2 update site to the composite update site would make our composite update site grow really fast in size. A single p2 repository for this example, including all dependencies is about 110Mb. A composite update site with just two p2 repositories would be 220Mb, and so on.

I think a good rule of thumb is

  • include all dependencies the first time we release our product’s update site (setting the property includeAllDependencies to true, and then setting it to false right after the first release)
  • for further releases do not include dependencies
  • include the dependencies again when we change the target platform of our product (indeed, Tycho will take the dependencies from our target platform)

Provide a command line installer

Now that our p2 composite repository is on the Internet, our users can simply download the zip file according to their OS, unzip it and enjoy it. But we could also provide another way for installing our custom Eclipse: an ant file so that the user will have to

The ant file will use the p2 director command line application to install our Eclipse product directly from the remote update site (the ant file is self-contained since if the director application is not already installed, it will install it as the first task).

Here’s the install.ant file (note that we ask the director to install our custom Eclipse product, customeclipse.example.ide and, explicitly, the main feature customeclipse.example.feature; this reflects what we specified in the product configuration, in particular, the fact that customeclipse.example.feature must be a ROOT feature, so that it can be updatable – see all the details in the previous post)

Note that this will always install the latest version present in the remote composite update site.

For instance, consider that you created zipped products for version 1.0.0, then you deployed a small upgrade only for your features, version 1.0.1, i.e., without releasing new zipped products. The ant script will install the custom Eclipse including version 1.0.1 of your features.

Some experiments

You may want to try and download the zipped product for your OS from this URL: https://sourceforge.net/projects/eclipseexamples/files/customeclipse/products/

After I deployed the self-contained p2 repository and the zipped products (activating the profiles release-ide-composite and deploy-ide-composite, with the property includeAllDependencies set to true, using the project customeclipse.example.ide.site), I deployed another p2 repository into the composite site only for the customeclipse.example.feature (activating the profiles release-composite and deploy-composite, i.e., using the project customeclipse.example.site).

Unzip the downloaded product, and check for updates (recall that the product is configured with the update site hosted on Sourceforge, through the p2.inf file described before). You will find that there’s an update for the Example Feature:

customeclipse before upgrading customeclipse available updates

After the upgrade and restart you should see the new version of the feature installed:

customeclipse after upgrading

Now, try to install the product using the ant file shown above, that can be downloaded from https://raw.githubusercontent.com/LorenzoBettini/customeclipse-example/master/customeclipse.example.tycho/install.ant.

You’ll have to wait a few minutes (and don’t worry about cookie warnings); run this version of the custom Eclipse, and you’ll find no available updates: check the installation details and you’ll see you already have the latest version of the Example Feature.

That’s all! Hope you find this post useful and… Happy Easter 🙂

Build your own custom Eclipse

In this tutorial I’ll show how to build a custom Eclipse distribution with Maven/Tycho. We will create an Eclipse distribution including our own features/plugins and standard Eclipse features, trying to keep the size of the final distribution small.

The code of the example can be found at: https://github.com/LorenzoBettini/customeclipse-example

First of all, we want to mimic the Eclipse SDK product and Eclipse SDK feature; have a look at your Eclipse Installation details

eclipse SDK installation details

You see that “Eclipse SDK” is the product (org.eclipse.sdk.ide), and “Eclipse Project SDK” is the feature (org.eclipse.sdk.feature.group).

Moreover, we want to deal with a scenario such that

Our custom feature can be installed in an existing Eclipse installation, thus we can release it independently from our custom Eclipse distribution. Our custom Eclipse distribution must be updatable, e.g., when we release a new version of our custom feature. 

The project representing our parent pom will be

  • customeclipse.example.tycho

The target platform is defined in

  • customeclipse.example.targetplatform

For this example we only need the org.eclipse.sdk feature and the native launcher feature

We created a plugin project and a feature project including such plugin (the plugin is nothing fancy, just an “Hello World Command” created with the Eclipse Plug-in project wizard):

  • customeclipse.example.plugin
  • customeclipse.example.feature

We also create another project for the p2 repository (Tycho packaging type: eclipse-repository) that distributes our plugin and feature (including the category.xml file)

  • customeclipse.example.site

All these projects are then configured with Maven/Tycho pom.xml files.

Then we create another feature that will represent our custom Eclipse distribution

  • customeclipse.example.ide.feature

This feature will then specify the features that will be part of our custom Eclipse distribution, i.e., our own feature (customeclipse.example.feature) and all the features taken from the Eclipse update sites that we want to include in our custom distribution.

Finally, we create another site project (Tycho packaging type: eclipse-repository) which is basically the same as customeclipse.example.site, but it also includes the product definition for our custom Eclipse product:

  • customeclipse.example.ide.site

NOTE: I’m using two different p2 repository projects because I want to be able to release my feature without releasing the product (see the scenario at the beginning of the post). This will also allow us to experiment with different ways of specifying the features for our custom Eclipse distribution.

Product Configuration

This is our product configuration file customeclipse.example.ide.product in the project customeclipse.example.ide.site and its representation in the Product Configuration Editor:

custom eclipse product configuration1

Note that we use org.eclipse.sdk.ide and org.eclipse.ui.ide.workbench for launching product extension identifier and application (we don’t have a custom application ourselves).

ATTENTION: Please pay attention to “uid” and “id” in the .product file, which correspond to “ID” and “Product” in the Product definition editor (quite confusing, isn’t it? 😉

This product configuration includes our customeclipse.example.ide.feature; we also inserted in the end the standard start level configuration, and other properties, like the standard workspace location.

The pom in this project will also activate the product materialization and archiving (we also specify the file name of the zip with our own pattern):

We chose NOT to include org.eclipse.example.ide.site as a module in our parent pom.xml: we include it only when we enable the profile build-ide: installing and provisioning a product takes some time, so you may not want to do that on every build invocation.  In that profile we add the customeclipse.example.ide.site module, this is the relevant part in our parent pom

In this profile, we also specify the environments for which we’ll build our custom Eclipse distribution. When this profile is not active, the target-platform-configuration will use only the current environment.

In the rest of the tutorial we’ll examine different ways of defining customeclipse.example.ide.feature. In my opinion, only the last one is the right one; but that depends on what you want to achieve. However, we’ll see the result and drawbacks of all the solutions.

You may want to try the options we detail in the following by cloning the example from https://github.com/LorenzoBettini/customeclipse-example and by modifying the corresponding files.

Include org.eclipse.sdk

The first solution is to simply include the whole org.eclipse.sdk feature in our customeclipse.example.ide.feature:

You can run the maven build specifying the profile build-ide

To get the materialized products (and the corresponding zipped versions).

NOTE: if you enable the tycho-source-feature-plugin in the parent pom to generate also source features, you’ll get this error during the build:

That’s because it tries to include in customeclipse.example.ide.feature.source the source feature of org.eclipse.sdk, which does not exist (org.eclipse.sdk already includes sources of its included features). You need to tell the tycho plugin to skip the source of org.eclipse.sdk:

The build should succeed.

Let’s copy the installed product directory (choose the one for your OS platform) to another folder; we perform the copy because a subsequent build will wipe out the target directory and we want to do some experiments. Let’s run the product and we see that our custom IDE shows our custom feature menu “Sample Menu” and the corresponding tool bar button:

If we check the installation details we see the layout mimicking the ones of Eclipse SDK (which is included in our product)

custom eclipse sdk installation details

Now let’s run the build again with above maven command.

If you have a look at the target directory you see that besides the products, in custom.eclipse.ide.site/target you also have a p2 repository,

custom ide site target

we will use the p2 repository to try and update the custom ide that we created in the first maven build (the one we copied to a different directory and that we ran in the previous step). So let’s add this built repository (in my case is /home/bettini/work/eclipse/tycho/custom-eclipse/customeclipse.example.ide.site/target/repository/) in the custom ide’s “Install New Software” dialog.

You see our Example Feature, and if you uncheck Group items by category you also see the Custom Eclipse Project SDK feature (corresponding to customeclipse.example.ide.feature) and Custom Eclipse SDK (corresponding to our product definition uid customeclipse.example.ide).

custom ide install new software 1 custom ide install new software 2

But wait… only the product is updatable! Why? (You see that’s the only one with the icon for updatable elements; if you try “Check for updates” that’s the only one that’s updatable)

Why can’t I update my “Example Feature” by itself?

If you try to select “Example Feature” in the “Install” dialog to force the update, and press Next…

custom ide install new software force 1

you’ll get an error, and the proposed solution, i.e., also update the product itself:

custom ide install new software force 2

And if you have a look at the original error…

custom ide install new software force 3

…you get an idea of the problem beneath: since we INCLUDED our “customeclipse.example.feature” in our product’s feature “customeclipse.example.ide.feature” the installed product will have a strict version requirement on “customeclipse.example.feature”: it will want exactly the version the original product was built with; long story short: you can’t update that feature, you can only update the whole product.

Before going on, also note in the target directory you have a zip of the p2 repository that has been created: customeclipse.example.ide.site-1.0.0-SNAPSHOT.zip it’s about 200 MB!  That’s because the created p2 repository contains ALL  the features and bundles INCLUDED in your product (which in our case, it basically means, all features INCLUDED in “customeclipse.example.ide.feature”).

Require org.eclipse.sdk

Let’s try and modify “customeclipse.example.ide.feature” so that it does NOT include the features, but DEPENDS on them (we can also set a version range for required features).

Let’s build the product.

First of all, note that the p2 repository zip in the target folder of customeclipse.example.ide.site is quite small!  Indeed, the repository contains ONLY our features, not all the requirements (in case, you can also force Tycho to include all the requirements), since, as stated above, the required feature will not be part of the repository.

Now let’s do the experiment once again:

  1. copy the built product for your OS into another directory
  2. run the product custom ide
  3. run another maven build
  4. add the new created p2 repository in the custom ide “Install new software” dialog

Well… the Example Feature does not appear as updatable, but this time, if we select it and press Next, we are simply notified that it is already installed, and that it will be updated

custom ide install new software force 4

So we can manually update it, but not automatically (“Check for updates” will still propose to update the whole product).

To make a feature updatable in our product we must make it a “Root level feature” (see also http://codeandme.blogspot.com/2014/06/tycho-11-install-root-level-features.html).

At the time of writing the Eclipse product definition editor does not support this feature, so we must edit the .product definition manually and add the line for specifying that customeclipse.example.feature must be a root level feature:

Let’s build again, note that this time the p2 director invocation explicitly installs customeclipse.example.feature

Let’s do the experiment again; but before trying to update let’s see that the installed software layout is now different: our Example Feature is now a root level feature (it’s also part of our Custom SDK IDE since it’s still required by customeclipse.example.ide.feature but that does not harm, and you may also want to remove that as a requirement in customeclipse.example.ide.feature).

custom eclipse sdk installation details 2

Hey! This time our “Example Feature” is marked as updatable

custom ide install new software 3

and also Check for updates proposes “Example Feature” as updatable independently from our product!

custom ide install new software 4

What happens if we make also customeclipse.example.ide.feature” a root feature? You may want to try that, and the layout of the installed software will list 3 root elements: our product “Custom Eclipse SDK”, our ide.feature “Custom Eclipse Project SDK” (which is meant to require all the software from other providers, like in this example, the org.eclipse.sdk feature itself) and our “Example Feature”.

This means that also “Custom Eclipse Project SDK” can be updated independently; this might be useful if we plan to release a new version of the ide.feature including (well, depending on) other software not included in Eclipse SDK itself (e.g., Mylyn, Xtext, or something else). At the moment, I wouldn’t see this as a priority so I haven’t set customeclipse.example.ide.feature as a root level feature in the product configuration.

Minimal Distribution

The problem of basing our distribution on org.eclipse.sdk is that the final product will include many features and bundles that you might not want in your custom distribution; e.g., CVS features, not to mention all the sources of the platform and PDE and lots of documentation. Of course, if that’s what we want, then OK. But if we want only the Java Development Tools in our custom distribution (besides our features of course)?

We can tweak the requirements in customeclipse.example.ide.feature and keep them minimal (note that the platform feature is really needed):

Build the product now.

Note also that the installed software has been reduced a lot:

custom eclipse minimal 4

The size of the zipped products dropped down to about 90Mb, instead of about 200Mb as they were before when we were using the whole org.eclipse.sdk feature.

However, by running this product you may notice that we lost some branding

  1. There’s no Welcome Page
  2. Eclipse starts with “Resource” Perspective, instead of “Java” Perspective
  3. Help => About (Note only “About” no more “About Eclipse SDK”) shows:

custom eclipse minimal 2

To recover the typical branding of Eclipse SDK, we have to know that such branding is implemented in the bundle org.eclipse.sdk (the bundle, NOT the homonymous feature).

So, all we have to do is to put that bundle in our feature’s dependencies

Rebuild, and try the product: we have all the branding back! 🙂

I hope you find this blog post useful 🙂

The sources of this example can be found here: https://github.com/LorenzoBettini/customeclipse-example

Publish an Eclipse p2 repository on Sourceforge with rsync

This can be seen as a follow-up post of my previous post on building Eclipse p2 composite repositories. In this blog post I’ll show an automatic way for publishing an Eclipse p2 (composite) repository (a.k.a. update site) on Sourceforge, using rsync for synchronization. You may find online many posts about publishing update sites on Github pages and recently on bintray. (as a reminder, rsync is a one-way synchronization tool, and we assume that the master replica is the one on sourceforge; rysnc, being a synchronization tool, will only transfer the changed files during synchronization).

I prefer sourceforge for some reasons:

  • you have full and complete access to the files upload system either with a shell or, most importantly for the technique I’ll describe here, with rsync. From what I understand, instead, bintray will manage the binary artifacts for you;
  • in order to create and update a p2 composite site you must have access to the current file system layout of the p2 update site, which I seem to understand is not possible with bintray;
  • you have download statistics and your artifacts will automatically mirrored in sourceforge’s mirrors.

By the way: you can store your git repository anywhere you want, and publish the binaries on sourceforge. (see this page and this other page).

I’ll reuse the same example of the previous post, the repository found here https://github.com/LorenzoBettini/p2composite-example, where you find all the mechanisms for creating and updating a p2 composite repository.

The steps of the technique I’ll describe here can be summarized as follows: when it comes to release a new child in the p2 composite update site (possibly already published on Sourceforge), the following steps are performed during the Maven/Tycho build

  1. Use rsync to get an update local version of the published p2 composite repository somewhere in your file system (this includes the case when you never released a version, so you’ll get a local empty directory)
  2. Build the p2 repository with Tycho
  3. Add the above created p2 repository as a new child in the local p2 composite repository (this includes the case where you create a new composite repository, since that’s your first release)
  4. Use rsync to commit the changes back to the remote p2 composite repository

Since we use rsync, we have many opportunities:

  • we’re allowed to manually modify (i.e., from outside the build infrastructure) the p2 composite repository, for instance by removing a child repository containing a wrong release, and commit the changes back;
  • we can release from any machine, notably from Jenkins or Hudson, since we always make sure to have a synchronized local version of the released p2 composite repository.

Prepare the directory on Sourceforge

This assumes that you have an account on Sourceforge, that you have registered a project. You need to create the directory that will host your p2 composite repository in the “Files” section.

For this example I created a new project eclipseexampleshttps://sourceforge.net/projects/eclipseexamples/, and I plan to store the p2 composite in the sourceforge file system on this path: p2composite.example/updates.

So I’ll create the directory structure accordingly (using the “Add Folder” button:

sourceforge create folder structure 1 sourceforge create folder structure 2 sourceforge create folder structure 3

Ant script for rsync

I’m using an ant script since it’s easy to call that from Maven, and also manually from the command line. This assumes that you have already rsync installed on your machine (or in the CI server from where you plan to perform releases).

This ant file is meant to be completely reusable.

Here’s the ant file

We have a macro for invoking rsync with the desired options (have a look at rsync documentation for understanding their meaning, but it should be straightforward to get an idea).

In particular, the transfer will be done with ssh, so you must have an ssh key pair, and you must have put the public key on your account on sourceforge. Either you created the key pair without a passphrase (e.g., for releasing from a CI server of your own), or you must make sure you have already unlocked the key pair on your local machine (e.g., with an ssh-agent, or with a keyring, depending on your OS).

The arguments source and dest will depend on whether we’re doing an update or a commit (see the two ant targets). If you define the property dryrun as -n then you can simulate the synchronization (both for update and commit); this is important at the beginning to make sure that you synchronize what you really mean to synchronize. Recall that when you perform an update, specifying the wrong local directory might lead to a complete deletion of that directory (the same holds for commit and the remote directory). Moreover, source and destinations URLs in rsync have a different semantics depending on whether they terminate with a slash or not, so make sure you understand them if you need to customize this ant file or to pass special URLs.

The properties rsync.remote.dir and rsync.local.dir will be passed from the Tycho build (or from the command line if you call the ant script directly). Once again, please use the dryrun property until you’re sure that you’re synchronizing the right paths (both local and remote).

Releasing during the Tycho build

Now we just need to call this ant’s targets appropriately from the Tycho build; I’ll do that in the pom.xml of the project that builds and updates the composite p2 repository.

Since I don’t want to push a new release on the remote site on each build, I’ll configure the plugins inside a profile (it’s up to you to decide when to release): here’s the new part:

Now the URL to access a remote path on sourceforge with ssh has the following shape

<username>,<project>@frs.sourceforge.net:/home/frs/project/<project>/<path>

So in my case I specified (again, the final / is crucial for what we want to synchronize with rsync, see the note above):

lbettini,eclipseexamples@frs.sourceforge.net:/home/frs/project/eclipseexamples/p2composite.example/updates/

The local URL specifies where the local p2 composite site is stored (see the previous post), in this example it defaults to

${user.home}/p2.repositories/updates/

Again, the final / is crucial.

We configured the maven-antrun-plugin with two executions:

  1. before updating the p2 composite update site (phase prepare-package) we make sure we have a synchronized local version of the repository
  2. after updating the p2 composite update site (phase verify) we commit the changes to the remote repository
  3. That’s all 🙂

Let’s try it

Of course, if you want to try it, you need a project on sourceforge and a directory on that project’s Files section (and you’ll have to change the URLs accordingly in the pom file).

To perform a release we need to call the build enabling the profile release-composite, and specify at least verify as goal:

Let’s say we still haven’t released anything.

Since the remote directory is empty, in our local file system we’ll simply have the directory created. In the end of the build, the composite site is created and the remote directory will be synchronized with our local contents: