Tag Archives: mirror

Mirror Eclipse p2 repositories with Tycho

I had previously written about mirroring Eclipse p2 repositories (see this blog tag), but I’ll show how to do that with Tycho and one of its plugins in this post.

The goal is always the same: speed up my Maven/Tycho builds that depend on target platforms and insulate me from external servers.

The source code of this example can be found here: https://github.com/LorenzoBettini/tycho-mirror-example.

I will show how to create a mirror of a few features and bundles from a few p2 repositories so that I can then resolve a target definition file against the mirror. In the POM, I will also create a version of the target definition file modified to use the local mirror (using Ant). Moreover, I will also use a Tycho goal to validate such a modified target definition file against the local mirror. The overall procedure is also automatized in the CI (GitHub Actions). This way, we are confident that we will create a mirror that can be used locally for our builds.

First of all, let’s see the target platform I want to use during my Maven/Tycho builds. The target platform definition file is taken from my project Edelta, based on Xtext.

As you see, it’s rather complex and relies on several p2 repositories. The last repository is the Orbit repository; although it does not list any installable units, that is still required to resolve dependencies of Epsilon (see the last but one location). We have to consider this when defining our mirroring strategy.

As usual, we define a few properties at the beginning of the POM for specifying the versions of the plugin and the parts of the p2 update site we will mirror from:

Let’s configure the Tycho plugin for mirroring (see the documentation of the plugin for all the details of the configuration):

The mirror will be generated in the user home subdirectory “eclipse-mirrors” (<destination> tag); we also define a few other mirroring options. Note that in this example, we cannot mirror only the latest versions of bundles (<latestVersionOnly>), as detailed in the comment in the POM. We also avoid mirroring the entire contents of the update sites (it would be too much). That’s why we specify single installable units. Remember that also dependencies of the listed installable units will be mirrored, so it is enough to list the main ones. You might note differences between the installable units specified in the target platform definition and those listed in the plugin configuration. Indeed, the target platform file could also be simplified accordingly, but I just wanted to have slight differences to experiment with.

If you write the above configuration in a POM file (a <packaging>pom</packaging> will be enough), you can already build the mirror running:

Remember that the mirroring process will take several minutes depending on your Internet connection speed since it will have to download about 500 Mb of data.

You can verify that all the specified repositories are needed to create the mirror correctly. For example, try to remove this part from the POM:

Try to create the mirror, and you should see this warning message because some requirements of Epsilon bundles cannot be resolved:

Those requirements are found in the Orbit p2 repository, which we have just removed for testing purposes.

Unfortunately, I found no way to make the build fail in such cases, even because it’s just a warning, not an error. I guess this is a limitation of the Eclipse mirroring mechanism. However, we will now see how to verify that the mirror contains all the needed software using another mechanism.

We create a modified version of our target definition file pointing to our local mirror. To do that, we create an Ant file (create_local_target.ant):

Note that this also handles path separators in Windows correctly. The idea is to replace lines of the shape <repository location=”https://…”/> with <repository location=”file:/…/eclipse-mirrors”/>. This file assumes the original target file is example.target, and the modified file is generated into local.target.

Let’s call this Ant script from the POM:

Finally, let’s use Tycho to validate the local.target file (see the documentation of the goal):

Now, if we run:

we build the mirror, and we create the local.target file.

Then, we can run the above goal explicitly to verify everything:

If this goal also succeeds, we managed to create a local mirror that we can use in our local builds. Of course, in the parent POM of your project, you must configure the build so that you can switch to local.target instead of using your standard .target file. (You might want to look at the parent POM of my Edelta project to take some inspiration.)

Since we should not trust a test that we never saw failing (see also my TDD book 🙂 let’s try to verify with the incomplete mirror that we learned to create by removing the Orbit URL. We should see that our local target platform cannot be validated:

Alternatively, let’s try to build our mirror with <latestVersionOnly>true</latestVersionOnly>, and during the validation of the target platform, we get:

In fact, we mirror only the latest version of org.antlr.runtime (4.7.2.v20200218-0804), which does not satisfy that requirement. That’s why we must use with <latestVersionOnly>false</latestVersionOnly> in this example.

For completeness, this is the full POM:

And this is the YAML file to build and verify in GitHub Actions:

I hope you found this post valuable, and happy mirroring! 🙂

Mirror Eclipse repositories from Eclipse

In a previous post, I blogged about mirroring Eclipse repositories with the p2.mirror Ant task; Since this Ant task needs to be run via the Eclipse antRunner application, you need a full installation of Eclipse on the machine that will run the task and I showed how to run this task from the command line.

It might be more comfortable sometimes to run it from Eclipse; for this, you need to create a launch configuration which runs an Eclipse Application, the same you run from the command line: org.eclipse.ant.core.antRunner.

We create a new general project (say, my.eclipse.mirror) and we create a mirror.ant file with the following example content (see the previous post for more details):

mirror-repos 2012-12-22 13:48:46

This example will mirror

  • org.eclipse.rcp.sdk, the eclipse launcher and the feature for adding update site features to your RCP, from the main Juno release site,
  • the whole Orbit repository (that version of the repository)
  • the whole Buckminster headless repository

Now we select the “Run Configurations…” dialog

mirror-repos 2012-12-22 13:49:02

and we create a new Eclipse Application launch configuration, which we call run_mirror; we choose, as the Program to Run, the application org.eclipse.ant.core.antRunner. Note that this application is installed by default in your SDK, but if there is a different target platform active in your workspace, which does not include the org.apache.ant bundle, this launch will fail.

mirror-repos 2012-12-22 13:49:45

In the Arguments tab, we must specify the -buildfile argument (expected by the runner) passing our ant file mirror.ant; this file is intended to be relative to the Working directory, thus we must modify it by selecting our project relative to the workspace:

mirror-repos 2012-12-22 13:50:35

Optionally, we save this launch configuration, using the “Common” tab, into our project

mirror-repos 2012-12-22 13:51:00

By default, the mirror will be saved into the folder eclipsemirror into your home folder; you may want to change this by specifying the additional program argument -Dtarget.dir=<path>, for instance,

mirror-repos 2012-12-22 13:51:27

Choose Apply and then Run; in your console view you should see that the mirror task has started

 

You can ignore possible problems of dependencies which cannot be satisfied

 

Depending on your Network connection it might take some time, but in the end you should see something like

mirror-repos 2012-12-22 15:39:46

And you can see that the mirror directory will mimic the original repositories structure

mirror-repos 2012-12-22 16:54:33

This example project can be found at https://github.com/LorenzoBettini/eclipse-mirror-example

Happy mirroring! 🙂

 

Mirror Eclipse repositories with p2.mirror Ant task

Inspired by this nice blog post, I decided it was time to try to mirror some Eclipse repositories to

  • speed up my builds that depend on target platforms
  • insulate myself from outside servers
  • be able to build also without an Internet connection
  • reduce network traffic 🙂

To do this I’m using p2 ant tasks for mirroring repositories.

When mirroring eclipse repositories, my main idea is to keep the path structure of the original repository URL. For instance, if you are using something like

http://download.eclipse.org/releases/juno

the mirror should differ only for the base url, e.g., something like

/home/myhome/eclipsemirror/releases/juno

This is useful if the p2 repositories you use for materializing your target platform are parametrized with respect to the base URL. For instance, I’m using Buckminster to materialize my target platforms (see also this post), and in my RMAP files I have something like

You see that the URLs are parametrized over the property eclipse.download (which defaults to http://download.eclipse.org). If I mirror those repositories keeping the same structure

then, switching to my local mirror for target materialization it’s just a matter of passing for the property eclipse.download the URL of my local directory, e.g., file:/home/bettini/eclipsemirror, without even changing my RMAP files.

So let’s start mirroring! We need to define an Ant script for the p2 antRunner.

For instance, for mirroring the whole orbit repository (with that particular drops version) we create this script, let’s call it mirror-orbit.xml:

Note that we keep in the target dir the same path structure of the original repository.

Since these Ant tasks need to be run via the Eclipse antRunner application, you need a full installation of Eclipse on the machine that will run the task. And you run this task with a command line like the following

Of course you can choose any target dir; the idea is however to always use the same target dir so that all repositories will be mirrored in that path.

Mirroring an entire repository might not always be the case, especially for Juno main release repository, which is quite huge. But you can specify in the Ant task the installable units you’re interested in; then, the p2 task will only mirror those installable units (and all its dependencies). For instance,

This task will mirror all the features that should let you define a target platform for RCP development with EMF and CDO.

NOTE: if you try to mirror org.eclipse.platform.sdk from the releases/juno repository, you will see that it will actually mirror the whole repository! (see also this forum post).

If you get some warnings during the mirror about unsolvable dependencies, you can ignore them: basically those dependencies are in a different repositories, and probably you will mirror those repositories too later.

Of course you can use several p2.mirror elements in the same Ant task. For example, this is the one we use in Emf Components, to have a mirror for our target platform: it also mirrors Swtbot and Xtext SDKs:

Final warning: it might take some time for the mirror task to complete (usually hours depending on your connection and download.eclipse.org load) and it will also take some hard disk space (for the above mirror it takes about 2 Gb).

You may have to experiment a bit to get all the features you need in the mirror; for instance, I didn’t know about the draw2d above, but I had to add it since during target materialization that feature was requested by some other feature. If you’re lost about that, you can always mirror the whole thing 😉

There’s also a follow up post showing how to run this ant task from Eclipse!

But then your builds will be faster 🙂