Tag Archives: rsync

Deploy your own custom Eclipse

This is the follow up of my previous post about building a custom Eclipse distribution. In this post I’ll show how to deploy the p2 site and the zipped products on Sourceforge. Concerning the p2 site, I’ll use the same technique, with some modifications, for building a composite update site and deploy it with rsync that I showed on another post.

In particular, we’ll accomplish several tasks:

  • creating and deploying the update site with only the features (without the products)
  • creating and deploying the update site including product definition and the zipped provisioned products
  • creating a self-contained update site (including all the dependencies)
  • providing an ant script for installing your custom Eclipse from the net

The code of the example can be found at: https://github.com/LorenzoBettini/customeclipse-example. In particular, I’ll start from where I left in the previous post.

The source code assumes a specific remote directory on Sourceforge, that is part of one of my Sourceforge projects, and it is writable only with my username and password. If you want to test this example, you can simply modify the property remote.dir in the parent pom specifying a local path in your computer (or by passing a value to the maven command with the syntax -Dremote.dir=<localpath>). Indeed, rsync can also synchronize two local directories.

Recall that when you perform a synchronization, specifying the wrong local directory might lead to a complete deletion of that directory. Moreover, source and destinations URLs in rsync have a different semantics depending on whether they terminate with a slash or not, so make sure you understand them if you need to customize this ant file or to pass special URLs.  

Creating and Deploying the p2 composite site

This part reuses most of what I showed in the previous posts:

In this blog post we want to be able to add a new p2 site to the composite update site (and deploy it) for two different projects:

  • customeclipse.example.site: This is the update site with only our features and bundles
  • customeclipse.example.ide.site: This is the update site with our features and bundles and the Eclipse product definition.

To reuse the ant files for managing the p2 composite update site and syncing it with rsync, and the Maven executions that use such ant files, we put the ant files in the parent project customeclipse.example.tycho, and we configure the Maven executions in the pluginManagement section of the parent pom.

We also put in the parent pom all the properties we’ll use for the p2 composite site and for rsync (again, please have a look at the previous posts for their meaning)

The pluginManagement section contains the configuration for managing the composite update site.

ATTENTION: in the following snipped, for the sake of readability, I split the <appArgLine> into several lines, but in your pom.xml it must be exactly in one (long) line.

The pluginManagement section also contains the configuration for updating and committing the composite update site to Sourceforge.

Now, we can simply activate such plugins in the build sections of our site projects described above.

In particular, we activate such plugins only inside profiles; for example, in the customeclipse.example.site project we have:

In customeclipse.example.ide.site we have similar sections, but the profiles are called differently, release-ide-composite and deploy-ide-composite, respectively.

So, if you want to update the p2 composite site with a new version containing only the features/bundles and deploy it on Sourceforge you need to run maven as follows

If you want to do the same, including the custom product definitions you need to run maven as follows (the additional build-ide profile is required because the customeclipse.example.ide.site is included as a Maven module only when that profile is activated; this way, products are created only when that profile is activated – just because provisioning a product requires some time and we don’t want to do that on normal builds)

NOTE: The remote directory on Sourceforge hosting  the composite update site will always be the same. This means that the local composite update site created and updated by both deploy-composite and deploy-ide-composite will be synchronized with the same remote folder.

In the customeclipse.example.ide.site, we added a p2.inf file with touchpoint instructions to add as update site in our Eclipse products the update site hosted on Sourceforge: http://sourceforge.net/projects/eclipseexamples/files/customeclipse/updates.

Deploying the zipped products

To copy the zipped products on Sourceforge we will still use rsync; actually, we won’t use any synchronization features: we only want to copy the zip files. I could have used the Ant Scp or Sftp tasks, but I experienced many problems with such tasks, so let’s use rsync also for that.

The ant file for rsync is slightly different with respect to the one shown in the previous post, since it has been refactored to pass the rsync macro more parameters. We still have the targets for update/commit synchronization; we added another target that will be used to simply copy something (i.e., the zipped products) to the remote directory, without any real synchronization. You may want to have a look at rsync documentation to fully understand the command line arguments.

In the customeclipse.example.ide.site, in the deploy-ide-composite profile, we configure another execution for the maven ant plugin (recall that in this profile the rsync synchronization configured in the parent’s pom pluginManagement section is also executed); this further execution will copy the zipped products to a remote folder on Sourceforge (as detailed in the previous post, you first need to create such folder using the Sourceforge web interface):

Note that when calling the rsync-copy-dir-contents of the rsync.ant file, we pass the properties as nested elements, in order to override their values (such properties’ value are already defined in the parent’s pom, and for this run we need to pass different values).

Now, if we run

many things will be executed:

  • rsync will synchronize our local composite update site with the remote composite update site
  • a new p2 site will be created, and added to our local composite update site
  • rsync will synchronize our local changes with the remote composite update site
  • Eclipse products will be created and zipped
  • the zipped products will be copied to Sourceforge

A self-contained p2 repository

Recall from the previous post that since in customeclipse.example.ide.feature we added Eclipse features (such as the platform and jdt) as dependencies (and not as included features), then the p2 update site we’ll create will not contain such features: it will contain only our own features and bundles. And that was actually intentional.

However, this means that the users of our features and of our custom Eclipse will still need to add the standard Eclipse update site before installing our features or updating the installed custom Eclipse.

If you want your p2 repository to be self-contained, i.e., to include also the external dependencies, you can do so by setting includeAllDependencies to true in the configuration of the tycho-p2-repository-plugin.

It makes sense to do that in the customeclipse.example.ide.site, so that all the dependencies for our custom Eclipse product will end up in the p2 repository:

However, doing so every time we add a new p2 update site to the composite update site would make our composite update site grow really fast in size. A single p2 repository for this example, including all dependencies is about 110Mb. A composite update site with just two p2 repositories would be 220Mb, and so on.

I think a good rule of thumb is

  • include all dependencies the first time we release our product’s update site (setting the property includeAllDependencies to true, and then setting it to false right after the first release)
  • for further releases do not include dependencies
  • include the dependencies again when we change the target platform of our product (indeed, Tycho will take the dependencies from our target platform)

Provide a command line installer

Now that our p2 composite repository is on the Internet, our users can simply download the zip file according to their OS, unzip it and enjoy it. But we could also provide another way for installing our custom Eclipse: an ant file so that the user will have to

The ant file will use the p2 director command line application to install our Eclipse product directly from the remote update site (the ant file is self-contained since if the director application is not already installed, it will install it as the first task).

Here’s the install.ant file (note that we ask the director to install our custom Eclipse product, customeclipse.example.ide and, explicitly, the main feature customeclipse.example.feature; this reflects what we specified in the product configuration, in particular, the fact that customeclipse.example.feature must be a ROOT feature, so that it can be updatable – see all the details in the previous post)

Note that this will always install the latest version present in the remote composite update site.

For instance, consider that you created zipped products for version 1.0.0, then you deployed a small upgrade only for your features, version 1.0.1, i.e., without releasing new zipped products. The ant script will install the custom Eclipse including version 1.0.1 of your features.

Some experiments

You may want to try and download the zipped product for your OS from this URL: https://sourceforge.net/projects/eclipseexamples/files/customeclipse/products/

After I deployed the self-contained p2 repository and the zipped products (activating the profiles release-ide-composite and deploy-ide-composite, with the property includeAllDependencies set to true, using the project customeclipse.example.ide.site), I deployed another p2 repository into the composite site only for the customeclipse.example.feature (activating the profiles release-composite and deploy-composite, i.e., using the project customeclipse.example.site).

Unzip the downloaded product, and check for updates (recall that the product is configured with the update site hosted on Sourceforge, through the p2.inf file described before). You will find that there’s an update for the Example Feature:

customeclipse before upgrading customeclipse available updates

After the upgrade and restart you should see the new version of the feature installed:

customeclipse after upgrading

Now, try to install the product using the ant file shown above, that can be downloaded from https://raw.githubusercontent.com/LorenzoBettini/customeclipse-example/master/customeclipse.example.tycho/install.ant.

You’ll have to wait a few minutes (and don’t worry about cookie warnings); run this version of the custom Eclipse, and you’ll find no available updates: check the installation details and you’ll see you already have the latest version of the Example Feature.

That’s all! Hope you find this post useful and… Happy Easter 🙂

Be Sociable, Share!

Publish an Eclipse p2 repository on Sourceforge with rsync

This can be seen as a follow-up post of my previous post on building Eclipse p2 composite repositories. In this blog post I’ll show an automatic way for publishing an Eclipse p2 (composite) repository (a.k.a. update site) on Sourceforge, using rsync for synchronization. You may find online many posts about publishing update sites on Github pages and recently on bintray. (as a reminder, rsync is a one-way synchronization tool, and we assume that the master replica is the one on sourceforge; rysnc, being a synchronization tool, will only transfer the changed files during synchronization).

I prefer sourceforge for some reasons:

  • you have full and complete access to the files upload system either with a shell or, most importantly for the technique I’ll describe here, with rsync. From what I understand, instead, bintray will manage the binary artifacts for you;
  • in order to create and update a p2 composite site you must have access to the current file system layout of the p2 update site, which I seem to understand is not possible with bintray;
  • you have download statistics and your artifacts will automatically mirrored in sourceforge’s mirrors.

By the way: you can store your git repository anywhere you want, and publish the binaries on sourceforge. (see this page and this other page).

I’ll reuse the same example of the previous post, the repository found here https://github.com/LorenzoBettini/p2composite-example, where you find all the mechanisms for creating and updating a p2 composite repository.

The steps of the technique I’ll describe here can be summarized as follows: when it comes to release a new child in the p2 composite update site (possibly already published on Sourceforge), the following steps are performed during the Maven/Tycho build

  1. Use rsync to get an update local version of the published p2 composite repository somewhere in your file system (this includes the case when you never released a version, so you’ll get a local empty directory)
  2. Build the p2 repository with Tycho
  3. Add the above created p2 repository as a new child in the local p2 composite repository (this includes the case where you create a new composite repository, since that’s your first release)
  4. Use rsync to commit the changes back to the remote p2 composite repository

Since we use rsync, we have many opportunities:

  • we’re allowed to manually modify (i.e., from outside the build infrastructure) the p2 composite repository, for instance by removing a child repository containing a wrong release, and commit the changes back;
  • we can release from any machine, notably from Jenkins or Hudson, since we always make sure to have a synchronized local version of the released p2 composite repository.

Prepare the directory on Sourceforge

This assumes that you have an account on Sourceforge, that you have registered a project. You need to create the directory that will host your p2 composite repository in the “Files” section.

For this example I created a new project eclipseexampleshttps://sourceforge.net/projects/eclipseexamples/, and I plan to store the p2 composite in the sourceforge file system on this path: p2composite.example/updates.

So I’ll create the directory structure accordingly (using the “Add Folder” button:

sourceforge create folder structure 1 sourceforge create folder structure 2 sourceforge create folder structure 3

Ant script for rsync

I’m using an ant script since it’s easy to call that from Maven, and also manually from the command line. This assumes that you have already rsync installed on your machine (or in the CI server from where you plan to perform releases).

This ant file is meant to be completely reusable.

Here’s the ant file

We have a macro for invoking rsync with the desired options (have a look at rsync documentation for understanding their meaning, but it should be straightforward to get an idea).

In particular, the transfer will be done with ssh, so you must have an ssh key pair, and you must have put the public key on your account on sourceforge. Either you created the key pair without a passphrase (e.g., for releasing from a CI server of your own), or you must make sure you have already unlocked the key pair on your local machine (e.g., with an ssh-agent, or with a keyring, depending on your OS).

The arguments source and dest will depend on whether we’re doing an update or a commit (see the two ant targets). If you define the property dryrun as -n then you can simulate the synchronization (both for update and commit); this is important at the beginning to make sure that you synchronize what you really mean to synchronize. Recall that when you perform an update, specifying the wrong local directory might lead to a complete deletion of that directory (the same holds for commit and the remote directory). Moreover, source and destinations URLs in rsync have a different semantics depending on whether they terminate with a slash or not, so make sure you understand them if you need to customize this ant file or to pass special URLs.

The properties rsync.remote.dir and rsync.local.dir will be passed from the Tycho build (or from the command line if you call the ant script directly). Once again, please use the dryrun property until you’re sure that you’re synchronizing the right paths (both local and remote).

Releasing during the Tycho build

Now we just need to call this ant’s targets appropriately from the Tycho build; I’ll do that in the pom.xml of the project that builds and updates the composite p2 repository.

Since I don’t want to push a new release on the remote site on each build, I’ll configure the plugins inside a profile (it’s up to you to decide when to release): here’s the new part:

Now the URL to access a remote path on sourceforge with ssh has the following shape

<username>,<project>@frs.sourceforge.net:/home/frs/project/<project>/<path>

So in my case I specified (again, the final / is crucial for what we want to synchronize with rsync, see the note above):

lbettini,eclipseexamples@frs.sourceforge.net:/home/frs/project/eclipseexamples/p2composite.example/updates/

The local URL specifies where the local p2 composite site is stored (see the previous post), in this example it defaults to

${user.home}/p2.repositories/updates/

Again, the final / is crucial.

We configured the maven-antrun-plugin with two executions:

  1. before updating the p2 composite update site (phase prepare-package) we make sure we have a synchronized local version of the repository
  2. after updating the p2 composite update site (phase verify) we commit the changes to the remote repository
  3. That’s all 🙂

Let’s try it

Of course, if you want to try it, you need a project on sourceforge and a directory on that project’s Files section (and you’ll have to change the URLs accordingly in the pom file).

To perform a release we need to call the build enabling the profile release-composite, and specify at least verify as goal:

Let’s say we still haven’t released anything.

Since the remote directory is empty, in our local file system we’ll simply have the directory created. In the end of the build, the composite site is created and the remote directory will be synchronized with our local contents:

Let’s have a look at the remote directory, it will contain the create p2 composite site

sourceforge uploaded artifacts 1

sourceforge uploaded artifacts 2

Let’s perform another release; Our local copy is up-to-date so we won’t receive anything during the update phase, but then we’ll commit another release

Let’s have a look at sourceforge and see the new release

sourceforge uploaded artifacts 3

Let’s remove our local copy and try to perform another release, this time the update phase will make sure our local composite repository is synchronized with the remote site (we’ll get the whole composite site we had already released), so that when we add another composite child we’ll update our local composite repository; then we’ll commit the changes to the server (again, by uploading only the modified files, i.e., the compositeArtifacts.xml and compositeContent.xml and the new directory with the new child repository:

Again, the remote site is correctly updated

sourceforge uploaded artifacts 4

Providing the URL of your p2 repository

Now that you have your p2 repository on sourceforge, you only need to give your users the URL to use for installing your features in Eclipse.

You have two forms for the URL

  • This will use the mirror infrastructure of sourceforge: http://sourceforge.net/projects/<project>/files/<path>
  • This will bypass mirrors: http://master.dl.sourceforge.net/project/<project>/<path>

If you use the mirror form, when installing in Eclipse (or provisioning a target platform) you’ll see warnings on the console of the shape

But it’s safe to ignore them.

For our example the URL can be one of the following:

  • With mirrors: http://sourceforge.net/projects/eclipseexamples/files/p2composite.example/updates/
  • Main site: http://master.dl.sourceforge.net/project/eclipseexamples/p2composite.example/updates/

You may want to try them both in Eclipse.

Please keep in mind that you may hit some unavailability errors now and then, if sourceforge sites are down for maintenance or unreachable for any reason… but that’s not much different when you hit a bad Eclipse mirror, or the main Eclipse download site is down… I guess no hosting site is perfect anyway 😉

I hope you find this blog post useful, Happy releasing! 🙂

 

Be Sociable, Share!