Mirror Eclipse repositories with p2.mirror Ant task

Inspired by this nice blog post, I decided it was time to try to mirror some Eclipse repositories to

  • speed up my builds that depend on target platforms
  • insulate myself from outside servers
  • be able to build also without an Internet connection
  • reduce network traffic 🙂

To do this I’m using p2 ant tasks for mirroring repositories.

When mirroring eclipse repositories, my main idea is to keep the path structure of the original repository URL. For instance, if you are using something like

http://download.eclipse.org/releases/juno

the mirror should differ only for the base url, e.g., something like

/home/myhome/eclipsemirror/releases/juno

This is useful if the p2 repositories you use for materializing your target platform are parametrized with respect to the base URL. For instance, I’m using Buckminster to materialize my target platforms (see also this post), and in my RMAP files I have something like

You see that the URLs are parametrized over the property eclipse.download (which defaults to http://download.eclipse.org). If I mirror those repositories keeping the same structure

then, switching to my local mirror for target materialization it’s just a matter of passing for the property eclipse.download the URL of my local directory, e.g., file:/home/bettini/eclipsemirror, without even changing my RMAP files.

So let’s start mirroring! We need to define an Ant script for the p2 antRunner.

For instance, for mirroring the whole orbit repository (with that particular drops version) we create this script, let’s call it mirror-orbit.xml:

Note that we keep in the target dir the same path structure of the original repository.

Since these Ant tasks need to be run via the Eclipse antRunner application, you need a full installation of Eclipse on the machine that will run the task. And you run this task with a command line like the following

Of course you can choose any target dir; the idea is however to always use the same target dir so that all repositories will be mirrored in that path.

Mirroring an entire repository might not always be the case, especially for Juno main release repository, which is quite huge. But you can specify in the Ant task the installable units you’re interested in; then, the p2 task will only mirror those installable units (and all its dependencies). For instance,

This task will mirror all the features that should let you define a target platform for RCP development with EMF and CDO.

NOTE: if you try to mirror org.eclipse.platform.sdk from the releases/juno repository, you will see that it will actually mirror the whole repository! (see also this forum post).

If you get some warnings during the mirror about unsolvable dependencies, you can ignore them: basically those dependencies are in a different repositories, and probably you will mirror those repositories too later.

Of course you can use several p2.mirror elements in the same Ant task. For example, this is the one we use in Emf Components, to have a mirror for our target platform: it also mirrors Swtbot and Xtext SDKs:

Final warning: it might take some time for the mirror task to complete (usually hours depending on your connection and download.eclipse.org load) and it will also take some hard disk space (for the above mirror it takes about 2 Gb).

You may have to experiment a bit to get all the features you need in the mirror; for instance, I didn’t know about the draw2d above, but I had to add it since during target materialization that feature was requested by some other feature. If you’re lost about that, you can always mirror the whole thing 😉

There’s also a follow up post showing how to run this ant task from Eclipse!

But then your builds will be faster 🙂

6 thoughts on “Mirror Eclipse repositories with p2.mirror Ant task

  1. Pingback: Mirror Eclipse repositories from Eclipse | Lorenzo Bettini

  2. Pingback: Building an Eclipse RCP Product with Buckminster | Lorenzo Bettini

  3. Mastermnd

    Hi Lorenzo, nice article. I was previously mirroring repos using both org.eclipse.equinox.p2.metadata.repository.mirrorApplication and org.eclipse.equinox.p2.artifact.repository.mirrorApplication.

    While these save me from having to create the ant script, I have found that your approach is more flexible since specific features/versions can be selected.

    I am currently trying to create a complete mirror of the latest version of the m2e-wtp project (https://www.eclipse.org/m2e-wtp/download/). Is there an easy way to determine the exact artifacts to include in the ant script?

    I can obviously mirror the metadata and then copy the latest version of all features to the ant script, but that process is very manual and annoying to repeat every time a new version is released. Is there a flag like “getLatestOnly” or something like that?

    Reply
  4. Lorenzo Bettini Post author

    Hi there
    if they have an sdk feature, probably mirroring that feature will include them all… otherwise you might want to try to mirror the whole http://download.eclipse.org/m2e-wtp/releases/juno/ for instance (if I understand your question correctly)

    However, I basically started to mirror repositories using the b3 aggregator, which is more reliable, allows to mirror from different repositories and also validate them all… as soon as I have some time I’ll blog about that as well 🙂

    Reply
  5. Pingback: Mirror Eclipse repositories from Eclipse | Lorenzo Bettini

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.