Materializing and Provisioning your Target Platform as local p2 site with Buckminster

This post was inspired by another blog post I had found on the web when trying to build a p2 site corresponding to a target platform defined for my projects. For an year I’ve been using Buckminster for building my Eclipse projects (especially when I had problems building my Xtext projects with Maven/Tycho, so that I decided to switch to Buckminster).

Having a p2 site for the current platform has some advantages, I guess:

  • Building RCP products will be easier (Buckminster relies on p2 directory, and all the bundles for building the product will be searched for in p2 sites);
  • It represents a (local) mirror for the target platform (for easier access).

In this post/tutorial I’ll describe my experiences

  • Define a target platform with Buckminster mechanisms (CSPEC, CQUERY and RMAP) instead of relying on target definitions with Eclipse Target Editor;
  • Build a p2 site with the contents of the so defined target platform.

In this tutorial we will create a target platform consisting of the following features

  • org.eclipse.rcp
  • org.eclipse.rcp.source
  • org.eclipse.emf.sdk
  • org.eclipse.equinox.executable

The code is available at: https://github.com/LorenzoBettini/Target_to_p2_with_Buckminster,

This tutorial will show how to materialize the target platform and build the p2 sites

  • in the IDE,
  • headlessly from the command line through ant, and
  • headlessly in Jenkins.

For the IDE you will need to install Buckminster main features in Eclipse, using the update site http://download.eclipse.org/tools/buckminster/updates-4.2.

NOTE: sometimes Buckminster’s cache in the IDE becomes stale, and you might experience errors about missing components; you might want to restart Eclipse and usually this makes the problem go away.

Why not the Target Editor?

I experienced many problems when using Eclipse Target Editor (based on p2 sites); I found it quite unreliable, especially for modifying a target definition, for updating a target definition (the Update button works 1 time out of 10); moreover, it takes some time for the target to be resolved (every time you reopen a target definition with the editor).

Furthermore, when you define a target for building RCP products, then you usually want to be able to build for several platforms, not only the current one; in these cases you have a checkbox to select (“Include all environments“):

But in that case, you must give up on “Include required software”, and this might get you into troubles about missing bundles, when building your product, if it needs to include more involved features (besides the standard “Eclipse RCP SDK” and “Eclipse Platform Launcher Executables”); for instance, if your product needs also EMF Eclipse Modeling Framework SDK (which is required when using Eclipse RCP 4.2).

Defining a target platform the “Buckminster way” does not have this limitation (and, honestly, once you get familiar with that, it’s much faster than using the target editor).

Defining a Target Platform with Buckminster

Although Buckminster is able to materialize a target platform starting from a standard target definition, it also provides an alternative way:

  1. Define a component which represents all the target features by creating a project and a custom CSPEC; this CSPEC basically declares the target features as dependencies
  2. Define a resource map (RMAP) which tells Buckminster where to find these dependencies; we will use of course official Eclipse p2 site repositories for these dependencies
  3. Define a component query (CQUERY) which materializes the component representing the target platform

In our example, this project is called bucky.example.rcpemf.target (it is a generic project without any nature). This is the project where we will define all Buckminster files (CSPEC, CQUERY and RMAP). In this project we define the component specification: buckminster.cspec

<?xml version="1.0" encoding="UTF-8"?>
<cs:cspec xmlns:cs="http://www.eclipse.org/buckminster/CSpec-1.0" name="bucky.example.rcpemf.target" componentType="buckminster" version="1.0.0">
    <cs:dependencies>
        <cs:dependency name="org.eclipse.emf.sdk" componentType="eclipse.feature"/>
        <cs:dependency name="org.eclipse.equinox.executable" componentType="eclipse.feature"/>
        <cs:dependency name="org.eclipse.rcp" componentType="eclipse.feature"/>
        <cs:dependency name="org.eclipse.rcp.source" componentType="eclipse.feature"/>
    </cs:dependencies>
</cs:cspec>

Here we use the name of the features for our target platform. (NOTE: org.eclipse.rcp and org.eclipse.rcp.source, together, correspond to Eclipse RCP SDK.)

The rmap (which we call target-platform.rmap) is defined as follows:

<?xml version="1.0" encoding="UTF-8"?>
<rm:rmap xmlns:rm="http://www.eclipse.org/buckminster/RMap-1.0"
	xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0">

	<rm:property key="projects.location" value="${workspace.root}" />

	<rm:searchPath name="juno-releases">
		<rm:provider componentTypes="osgi.bundle,eclipse.feature"
			readerType="p2" source="false" mutable="false">
			<rm:uri format="http://download.eclipse.org//releases/juno" />
		</rm:provider>
	</rm:searchPath>
	<rm:searchPath name="juno-updates">
		<rm:provider componentTypes="osgi.bundle,eclipse.feature"
			readerType="p2" source="false" mutable="false">
			<rm:uri format="http://download.eclipse.org/eclipse/updates/4.2" />
		</rm:provider>
	</rm:searchPath>
	<rm:searchPath name="local-source">
		<rm:provider componentTypes="eclipse.feature,osgi.bundle,buckminster"
			readerType="local" mutable="false">
			<rm:property key="buckminster.mutable" value="false" />
			<rm:uri format="{0}/{1}">
				<bc:propertyRef key="projects.location" />
				<bc:propertyRef key="buckminster.component" />
			</rm:uri>
		</rm:provider>
	</rm:searchPath>
	<rm:locator searchPathRef="local-source" failOnError="false" />
	<rm:locator searchPathRef="juno-releases" failOnError="false" />
	<rm:locator searchPathRef="juno-updates" failOnError="true" />
</rm:rmap>

In our case we use the two main Eclipse p2 repositories (if the features you need are available from other p2 repositories, you will need to provide p2 reader types for these repositories as well). We will not use regular expressions for specifying the source p2 repositories, since there are no ambiguities in this example; instead, we simply specify the locators in the order we desire (preferring juno release site to juno updates). Note that we also have a search path for local sources. This is useful for building headlessly, so that Buckminster is able to find, resolve and bind to the workspace our own projects.

Finally, the component query (which we call target-platform.cquery) is defined as

<?xml version="1.0" encoding="UTF-8"?>
<cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="target-platform.rmap">
    <cq:rootRequest name="bucky.example.rcpemf.target" componentType="buckminster"/>
    <cq:property key="target.arch" value="*"/>
    <cq:property key="target.os" value="*"/>
    <cq:property key="target.ws" value="*"/>
</cq:componentQuery>

Thus we request the component corresponding to our project bucky.example.rcpemf.target. Note that we specify that we are not interested in any specific architecture, so that we will have a target platform for all platforms/environments/architectures. Note also that in the cquery we reference the above defined rmap.

Materializing the Target Platform

We can now materialize the target platform so defined by opening the target-platform.cquery with Buckminster corresponding editor:

WAIT: If you pressed the “Resolve and Materialize” now, Buckminster would materialize the target platform in the directory tp of a hidden project in your workspace (.buckminster). Moreover, by default, the created target platform is based on the configuration of the currently running Eclipse instance. If you want to start from a plain and empty platform (just like you do with a standard target definition with the target editor), there are a few more steps to perform:

  1. Create an empty target platform manually that contains one single and empty directory.
  2. Set this target platform active.
  3. A subsequent resolution/materialization will use that platform.

You can specify any directory of your filesystem, in any case, make sure the directory you choose already exists.

One possible way of doing this, as illustrated also here, is

  1. Create a new general project named TP (or some name of your preference) in the workspace
  2. In “Window” => “Preferences” => “Plug-in Development” => “Target Platform”
    Select Add…
  3. Start with an empty target definition
  4. Enter TP in the Name: field (or some name of your preference)
  5. Add a directory
  6. Click on “Variables…” scroll down and select “workspace_loc” and then type TP in the Argument: field

This procedure is shown by the following screenshots.

NOTE: these additional steps are not required when building headlessly

Once you pressed Finish and set this new target as the active target platform, you’re ready to materialize the cquery from the cquery editor. Of course, it might take some time for downloading all the features. Once the materialization is finished, if you followed the above procedure and created a general project TP in your workspace, you can see the materialized target platform

Create a feature for the p2 site for your target platform

Once the target platform is resolved (you can also have a look at the current new target platform from the Eclipse preferences), we are ready to create a feature that we use to create the p2 site corresponding to our target platform.

We thus create a new feature project, that we call bucky.example.rcpemf.target.site, and we initialize it with all the bundles (plugins and fragments) of the current target platform (if you have other bundle projects in your workspace, make sure to unselect such additional workspace projects)

Before creating the p2 site, you may want to create a buckminster.properties (e.g., in the main project bucky.example.rcpemf.target) where we set some properties for Buckminster (e.g., that we want to build a site for all environments, and that we want the site to be created in a specific folder, in this example /tmp/bucky):

# Where all the output should go
buckminster.output.root=/tmp/bucky/build
# Where the temp files should go
buckminster.temp.root=/tmp/bucky
# How .qualifier in versions should be replaced
qualifier.replacement.*=generator:lastRevision

target.os=*
target.ws=*
target.arch=*

We are now ready to build a p2 site for this feature with Buckminster:

  1. right click on the bucky.example.rcpemf.target.site project
  2. select Buckminster -> Invoke Action…
  3. select the action site.p2 (and, additionally, select the buckminster.properties file above)


If you used the same property file, you can find the site.p2 in /tmp/bucky/build/bucky.example.rcpemf.target.site_1.0.0-eclipse.feature/site.p2/ .

The p2 site you’ve just created could already be used, for instance, for building your RCP product with Buckminster through the p2 director. However, as it is, it contains only bundles, no (installable) features, thus it cannot be used, for instance, as a mirror for your target platform… see the next section

Create a feature for the p2 MIRROR site for your target platform

We create a new feature project bucky.example.rcpemf.target.mirror.site, and, as before, we initialize it with all the bundles (plugins and fragments) of the current target platform (if you have other bundle projects in your workspace, make sure to unselect such additional workspace projects); but this time, once the feature project is created, we also add to the feature.xml all the features that we used to create our target platform:

Let’s run the action site.p2 (using the same buckminster.properties file) on this new feature project. This time, we will end up with a p2 site with all the installable features of our target platform! You can verify this by using the Eclipse install new features menu and using as repository the one just created (if you used the same output directory it will be located at /tmp/bucky/build/bucky.example.rcpemf.target.mirror.site_1.0.0-eclipse.feature/site.p2/).

Even more, you might create a target definition file, open it with the Eclipse target editor, and create a target definition using your local mirror!

Headless execution from the command line

If you want to materialize the target platform and build the p2 site from the command line, you need to install the headless version of Buckminster.

For instance, you download the director from http://www.eclipse.org/buckminster/downloads.html and then you run the director as follows

director -r \
   http://download.eclipse.org/tools/buckminster/headless-4.2/ \
   -d <WHERE_TO_INSTALL> -p Buckminster \
   -i org.eclipse.buckminster.cmdline.product \
   -i org.eclipse.buckminster.core.headless.feature.feature.group \
   -i org.eclipse.buckminster.pde.headless.feature.feature.group

Headlessly, we will need to

  1. resolve the cquery for the target platform (that we already showed above);
  2. resolve the cquery for the site feature, (in this case, it is the one for the mirror site), mirror-project.cquery:
    <?xml version="1.0" encoding="UTF-8"?>
    <cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="target-platform.rmap">
        <cq:rootRequest name="bucky.example.rcpemf.target.mirror.site" componentType="eclipse.feature"/>
        <cq:property key="target.arch" value="*"/>
        <cq:property key="target.os" value="*"/>
        <cq:property key="target.ws" value="*"/>
    </cq:componentQuery>

    Note we reuse the same rmap we saw before. This way, during the materialization of the mirror site feature, possible additional dependencies will be resolved by Buckminster and added to the target platform;

  3. perform the site.p2 action (before we also perform a buckminster.clean) on the component of the mirror site

We then write an ANT file, build.ant, in the project bucky.example.rcpemf.target, as follows (this was inspired by the one generated by the Xtext wizard, developed by Dennis HübnerI’ve blogged about that, and from the one that can be found here):

<?xml version="1.0" encoding="UTF-8"?>
<!--
     Buckminster Headless - build

     buckminster.home must be specified on the command line, e.g.,
         ant -Dbuckminster.home=/home/bettini/buckminster -f build.ant

     Properties:
         WORKSPACE              Eclipse workspace location, or hudson job workspace
         build.root             Where to build? WARNING: This folder will be cleaned up, so do not point to user.home or something important
                            Default: ${WORKSPACE}/buildroot
        buckminster.home    Buckminster headless to use. See http://www.eclipse.org/buckminster/downloads.html
        projects.location    Where to find projects to build?
                            Default: ${WORKSPACE}
-->
<project name="Buckminster Headless" default="buckminster">
    <property name="WORKSPACE" location="${ant.file}/../../" />
    <property name="build.root" location="${WORKSPACE}/buildroot" />
    <property name="projects.location" location="${WORKSPACE}" />
    <property name="resolve.commands.file" location="${projects.location}/bucky.example.rcpemf.target/headless-resolve-commands.txt" />
    <property name="perform.commands.file" location="${projects.location}/bucky.example.rcpemf.target/headless-perform-commands.txt" />
    <property name="target.platform" value="target.platform" />

    <target name="buckminster" depends="cleanup" description="description">
        <fail unless="buckminster.home" message="buckminster.home must be specified." />

        <echo message="IMPORTANT: Populating an empty target platform may took over 10 minutes." />

        <java fork="true" dir="${buckminster.home}" logError="true" classname="org.eclipse.core.launcher.Main" failonerror="true">
            <classpath>
                <fileset dir="${buckminster.home}/plugins">
                    <include name="org.eclipse.equinox.launcher_*.jar" />
                </fileset>
            </classpath>
            <arg line='-update' />
            <arg line='-data "${build.root}/buckminster.workspace"' />
            <arg line='-configuration "${build.root}/configuration"' />
            <arg line='--script "${resolve.commands.file}"' />
            <sysproperty key="projects.location" value="${projects.location}" />
            <sysproperty key="buckminster.output.root" value="${build.root}/buckminster.output" />
            <sysproperty key="buckminster.temp.root" value="${build.root}/buckminster.temp" />
            <sysproperty key="target.platform" value="${build.root}/${target.platform}" />
            <jvmarg line=" -Xms256m -Xmx512m" />
        </java>

        <java fork="true" dir="${buckminster.home}" logError="true" classname="org.eclipse.core.launcher.Main" failonerror="true">
            <classpath>
                <fileset dir="${buckminster.home}/plugins">
                    <include name="org.eclipse.equinox.launcher_*.jar" />
                </fileset>
            </classpath>
            <arg line='-update' />
            <arg line='-data "${build.root}/buckminster.workspace"' />
            <arg line='-configuration "${build.root}/configuration"' />
            <arg line='--script "${perform.commands.file}"' />
            <sysproperty key="projects.location" value="${projects.location}" />
            <sysproperty key="buckminster.output.root" value="${build.root}/buckminster.output" />
            <sysproperty key="buckminster.temp.root" value="${build.root}/buckminster.temp" />
            <sysproperty key="target.platform" value="${build.root}/${target.platform}" />
            <jvmarg line=" -Xms256m -Xmx512m" />
        </java>

        <echo message=" " />
        <echo message="Updatesite output in: ${build.root}/buckminster.output/bucky.example.rcpemf.target.mirror.site_*-eclipse.feature/site.p2/" />
    </target>

    <target name="cleanup">
        <delete failonerror="false" includeemptydirs="true">
            <fileset dir="${build.root}" defaultexcludes="false">
                <exclude name="**/.metadata/.plugins/org.eclipse.pde.core/.bundle_pool/" />
                <exclude name="**/${target.platform}/" />
            </fileset>
        </delete>
    </target>

    <target name="reset.target-platform">
        <delete includeemptydirs="true">
            <fileset dir="${build.root}" defaultexcludes="false">
                <include name="**/.metadata/.plugins/org.eclipse.pde.core/.bundle_pool/" />
                <include name="**/${target.platform}/" />
            </fileset>
        </delete>
    </target>
</project>

You will need to run ant passing this file and also specifying the path where your headless Buckminster is installed, in my case I run

ant -Dbuckminster.home=/home/bettini/buckminster -f build.ant

This ant script runs Buckminster twice:

  1. The first time it runs the materialization commands (stored in the file bucky.example.rcpemf.target/headless-resolve-commands.txt)
    setpref targetPlatformPath="${target.platform}"
    resolve "${projects.location}/bucky.example.rcpemf.target/target-platform.cquery"
    resolve "${projects.location}/bucky.example.rcpemf.target/mirror-project.cquery"
  2. Then it runs the actual commands to build the p2 site (stored in the file bucky.example.rcpemf.target/headless-perform-commands.txt); in this case we only build the p2 site for the mirror feature
    clean
    build
    perform -D target.os=* -D target.ws=* -D target.arch=* "bucky.example.rcpemf.target.mirror.site#buckminster.clean"
    perform -D target.os=* -D target.ws=* -D target.arch=* "bucky.example.rcpemf.target.mirror.site#site.p2"

You need to execute Buckminster twice because of some conflicting bundles (see the discussion here) which cause problems if you attempt to run everything in one shot; if you did not separate the steps, you would get something like

INFO:  perform '-D' 'target.os=*' '-D' 'target.ws=*' '-D' 'target.arch=*' 'bucky.example.rcpemf.target.site#site.p2'
CSpec org.apache.lucene.core:osgi.bundle$3.5.0.v20120319-2345 has no  action, group, or local artifact named bundle.and.fragments

Once the script is finished you will get the p2 site in the subdirectory buildroot/buckminster.output/bucky.example.rcpemf.target.mirror.site_*-eclipse.feature/site.p2.

Headless execution in Jenkins

The last task is to build headlessly in Jenkins; of course, this assumes you have a working Jenkins installation with the Buckminster plugin (I’ve blogged about that also).

We write the commands to execute in two text files (similar to the ones shown in the previous section):

  • bucky.example.rcpemf.target/jenkins-resolve-mirror-commands.txt
    resolve 'bucky.example.rcpemf.target/target-platform.cquery'
    resolve 'bucky.example.rcpemf.target/project.cquery'
  • bucky.example.rcpemf.target.mirror.site/jenkins-perform-commands.txt
    clean
    build
    perform -D target.os=* -D target.ws=* -D target.arch=* bucky.example.rcpemf.target.mirror.site#buckminster.clean
    perform -D target.os=* -D target.ws=* -D target.arch=* bucky.example.rcpemf.target.mirror.site#site.p2

Then we can create a Jenkins job, which fetches the sources from the git repository

specify two Buckminster execution steps (as noted in the previous section, this is mandatory for the whole process to succeed):

and archive the artifacts

When the job finishes, you can see the archived p2 site

That’s all…

I hope you enjoyed this post, and comments are more than welcome :)

Be Sociable, Share!

9 thoughts on “Materializing and Provisioning your Target Platform as local p2 site with Buckminster

    1. Lorenzo Bettini Post author

      Hi Vlad, glad you liked it :)
      usually for non-RCP stuff it is easier: you do not need the executable features and the site.p2 usually is not architecture dependent… so I would say, no, there should be no difference… are you experiencing problems in your context?

      Reply
      1. Vlad

        No, I don’t have any problems directly, but I got it working in an ad-hoc fashion for a year or so ago and since then didn’t dared to change anything for fear that it will start misbehaving. I’m using a static target platform, manually installed, and I would like very much to automate that. Hopefully I can get it to work with the walkthrough above.

        Reply
  1. Pingback: Building Xtext projects with Buckminster | Lorenzo Bettini

  2. Pingback: Mirror Eclipse repositories with p2.mirror Ant task | Lorenzo Bettini

  3. Pingback: Building an Eclipse RCP Product with Buckminster | Lorenzo Bettini

  4. Boris Brodski

    Thank you for the great tutorial!

    In general I share your opinion on the “Target Editor”, but I found a way to make it do it’s job.

    - open .target file with the eclipse text editor
    - change, what you want to change
    - increase “sequenceNumber” in the “target” tag
    - save the file

    Eclipse automatically resolves new target platform.

    If I what to add new features, I add those in the “Target Editor” itself, inspects changes (diff) and then
    reapply changes manually using the method described above.

    Reply
    1. Lorenzo Bettini Post author

      Hi Boris
      I’m basically doing the same procedure when I use a .target definition file… but it’s a kind of shame that the UI tooling about target definition files is so poor that it’s mostly unusable :)

      Reply

Leave a Reply