Monthly Archives: April 2022

Mirror Eclipse p2 repositories with Tycho

5 Replies

I had previously written about mirroring Eclipse p2 repositories (see this blog tag), but I’ll show how to do that with Tycho and one of its plugins in this post.

The goal is always the same: speed up my Maven/Tycho builds that depend on target platforms and insulate me from external servers.

The source code of this example can be found here: https://github.com/LorenzoBettini/tycho-mirror-example.

I will show how to create a mirror of a few features and bundles from a few p2 repositories so that I can then resolve a target definition file against the mirror. In the POM, I will also create a version of the target definition file modified to use the local mirror (using Ant). Moreover, I will also use a Tycho goal to validate such a modified target definition file against the local mirror. The overall procedure is also automatized in the CI (GitHub Actions). This way, we are confident that we will create a mirror that can be used locally for our builds.

First of all, let’s see the target platform I want to use during my Maven/Tycho builds. The target platform definition file is taken from my project Edelta, based on Xtext.

As you see, it’s rather complex and relies on several p2 repositories. The last repository is the Orbit repository; although it does not list any installable units, that is still required to resolve dependencies of Epsilon (see the last but one location). We have to consider this when defining our mirroring strategy.

As usual, we define a few properties at the beginning of the POM for specifying the versions of the plugin and the parts of the p2 update site we will mirror from:

Let’s configure the Tycho plugin for mirroring (see the documentation of the plugin for all the details of the configuration):

The mirror will be generated in the user home subdirectory “eclipse-mirrors” (<destination> tag); we also define a few other mirroring options. Note that in this example, we cannot mirror only the latest versions of bundles (<latestVersionOnly>), as detailed in the comment in the POM. We also avoid mirroring the entire contents of the update sites (it would be too much). That’s why we specify single installable units. Remember that also dependencies of the listed installable units will be mirrored, so it is enough to list the main ones. You might note differences between the installable units specified in the target platform definition and those listed in the plugin configuration. Indeed, the target platform file could also be simplified accordingly, but I just wanted to have slight differences to experiment with.

If you write the above configuration in a POM file (a <packaging>pom</packaging> will be enough), you can already build the mirror running:

Remember that the mirroring process will take several minutes depending on your Internet connection speed since it will have to download about 500 Mb of data.

You can verify that all the specified repositories are needed to create the mirror correctly. For example, try to remove this part from the POM:

Try to create the mirror, and you should see this warning message because some requirements of Epsilon bundles cannot be resolved:

Those requirements are found in the Orbit p2 repository, which we have just removed for testing purposes.

Unfortunately, I found no way to make the build fail in such cases, even because it’s just a warning, not an error. I guess this is a limitation of the Eclipse mirroring mechanism. However, we will now see how to verify that the mirror contains all the needed software using another mechanism.

We create a modified version of our target definition file pointing to our local mirror. To do that, we create an Ant file (create_local_target.ant):

Note that this also handles path separators in Windows correctly. The idea is to replace lines of the shape <repository location=”https://…”/> with <repository location=”file:/…/eclipse-mirrors”/>. This file assumes the original target file is example.target, and the modified file is generated into local.target.

Let’s call this Ant script from the POM:

Finally, let’s use Tycho to validate the local.target file (see the documentation of the goal):

Now, if we run:

we build the mirror, and we create the local.target file.

Then, we can run the above goal explicitly to verify everything:

If this goal also succeeds, we managed to create a local mirror that we can use in our local builds. Of course, in the parent POM of your project, you must configure the build so that you can switch to local.target instead of using your standard .target file. (You might want to look at the parent POM of my Edelta project to take some inspiration.)

Since we should not trust a test that we never saw failing (see also my TDD book 🙂 let’s try to verify with the incomplete mirror that we learned to create by removing the Orbit URL. We should see that our local target platform cannot be validated:

Alternatively, let’s try to build our mirror with <latestVersionOnly>true</latestVersionOnly>, and during the validation of the target platform, we get:

In fact, we mirror only the latest version of org.antlr.runtime (4.7.2.v20200218-0804), which does not satisfy that requirement. That’s why we must use with <latestVersionOnly>false</latestVersionOnly> in this example.

For completeness, this is the full POM:

And this is the YAML file to build and verify in GitHub Actions:

I hope you found this post valuable, and happy mirroring! 🙂

Multibooting with GRUB

10 Replies

4th July, updated with BTRFS installations.

There’s also a more recent and simpler version of this post.

Besides Windows (which I rarely use) on my computers, I have a few Linux distributions. Grub 2 does a good job booting Windows and Linux, especially thanks to os-prober, in autodetecting other operating systems in other partitions of the same computer. However, there are a few “buts” in this strategy:

  1. Typically, the last installed Linux distribution, say L1, installs its own grub as the main one, and when you upgrade the kernel in another Linux distribution, say L2, you have to boot into L1 and “update-grub” so that the main grub configuration learns about the new kernel of L2. Only then can you boot the new kernel of L2. Of course, you can change the main grub by reordering the EFI entries, e.g., by using the computer’s BIOS, but again, that’s far from optimal.
  2. Not all Linux distributions’ grub configurations can boot other Linux distributions. For example, Arch-based distros like EndeavourOS and Manjaro can boot Ubuntu-based distros, but not the other way around (unless you fix a few things in the grub configuration of Ubuntu)! Recently, I also started to use Fedora. I found out that os-prober in Ubuntu and EndeavourOS does not detect the configurations correctly to boot Fedora: recently, Fedora switched to “blscfg” (https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault), and as a result, Ubuntu and EndeavourOS create grub configurations that do not consider the changes you made in Fedora’s /etc/default/grub.

That’s why I started to experiment with grub configurations. I still have a “main grub” in a Linux installation, which simply “delegates” to the grub configurations of the other Linux installations. This way, I can solve both the problems above!

In this blog post, I’ll show how I did that. Note that this assumes you use EFI to boot.

I have Windows 10, Kubuntu, EndeavourOS, and Fedora on the same computer in this example. I will configure the grub installation of Fedora so that it delegates to Windows, Kubuntu, and EndeavourOS without relying on os-prober.

This is the disk layout of my computer so that you understand the numbers in the grub configuration that I’ll show later (I omit other partitions like Windows recovery).

The key point is modifying the file /etc/grub.d/40_custom. I guess you already know that you should not modify directly grub.cfg, because a system update or a grub update (e.g., “update-grub”) will overwrite that file.

The file /etc/grub.d/40_custom already has some contents that must be left as they are: you add your lines after the existing ones. For example, in Fedora, you have:

We will use the option configfile of grub configuration (see https://www.gnu.org/software/grub/manual/grub/grub.html#configfile): “Load file as a configuration file. If file defines any menu entries, then show a menu containing them immediately.” The Arch wiki also explains it well:

If the other distribution has already a valid /boot folder with installed GRUB, grub.cfg, kernel and initramfs, GRUB can be instructed to load these other grub.cfg files on-the-fly during boot.

The idea is to put in /etc/grub.d/40_custom an entry for each Linux distribution, pointing to the grub.cfg of that distribution after setting the root partition. Thus, the path to the grub.cfg must be intended as an absolute path in that partition. If you look at the partition numbers above, these are the two entries for booting EndeavourOS and Kubuntu:

NOTE: the “rmmod tpm” is required to avoid TPM errors when booting those systems (“Unknown TPM error”, “you need to load the kernel first”). It happened on my Dell XPS 13, for example. Adding that line (i.e., not loading the module “tpm”) solved the problem.

Remember that the path assumes that the /boot directory is not mounted on a separate partition. If, instead, that’s the case, you probably have to remove “/boot”, but I haven’t tried that.

Concerning the entry for Windows, here it is:

In this entry, the root must correspond to the EFI partition, NOT to the partition of Windows.

Save the file and regenerate the grub configuration. In other Linux distributions, it would be a matter of running “update-grub,” but in Fedora, it is:

Now reboot, and you should see the grub menu of Fedora and then, at the bottom, the entries for EndeavourOS, Kubuntu, and Windows. Choosing “EndeavourOS” or “Kubuntu” will NOT boot directly in these systems: it will show the grub menu of “EndeavourOS” or “Kubuntu.”

If you upgrade the kernel on one of these two systems, their grub configuration will be correctly updated. There’s no need to boot into Fedora to update its grub configuration 🙂

If you want to configure the grub in another Linux distribution, please remember that Fedora stores the grub.cfg in /boot/grub2 instead of /boot/grub, so you should write the entry for Fedora with the right path. However, if you plan to boot Fedora with this mechanism, you should disable “blscfg” in the Fedora grub configuration, or you will not be able to boot Fedora (errors “increment.mod” and “blscfg.mod” not found).

Now that we verified that it works, we can remove the entries generated by os-prober. In /etc/default/grub add the line:

and regenerate the grub configuration.

If you want Grub to remember the last choice, you can look at this post.

On a side note, due to the way Fedora uses grub (https://fedoraproject.org/wiki/Changes/HiddenGrubMenu), without os-prober, you will not see the grub menu unless you press ESC. After the timeout, it will simply boot on the default entry. To avoid that and see the grub menu, just run:

And the grub menu will get back as usual.

Then, you can also remove os-prober from the other Linux installations since it is useless now.

These were the original grub menus of Fedora and EndeavourOS before applying the modifications described in this post:

Pretty crowded!

This is the result after the procedure described in this post (note that from the Fedora grub menu, you select EndeavourOS to land its grub menu and Kubuntu to land its grub menu):

Much better! 🙂

If you need to boot an installation in a BTRFS filesystem (which also includes the /boot directory and the grub.cfg), things are slightly more complex. In fact, BTRFS installations are typically based on subvolumes. The root subvolume is typically denoted by the label “@”. This must be taken into consideration when creating the menu entry.

For example, I’ve also installed Arch on my computer using BTRFS, and the root subvolume is denoted by “@”. The menu entry is as follows:

Note the presence of “/@” in the configfile specification.

That’s not all. If the GRUB configuration specified in configfile has submenus, for example, automatically generated by grub-btrfs, you also must define the prefix variable appropriately (in fact, grub-btrfs generates entries relying on such a variable):