Tag Archives: xtext

Oomph setup for Xtext projects

In this blog post I’ll describe my experience in preparing an Oomph setup for a non-trivial Xtext project, Xsemantics.

This setup was kind of challenging because of the following features of my project, but I guess most of them can be found in any Xtext project:

  • generated sources are not stored in the Git repository (these include Xtend generated Java files and Java files generated during the MWE2 workflow)
  • the MWE2 workflow(s) must be run during the workspace setup (I have several DSLs in this project)
  • one of the DSL “inherits” from another DSL, so when running the MWE2 of the inheriting DSL the parent DSL must have already been built (i.e., Java classes must be compiled)

I hope this post can be useful for other Xtext developers.

This will not be a tutorial: it will be a collection of hints and procedures for preparing the final setup which can be found here: https://github.com/LorenzoBettini/xsemantics/blob/master/devtools/it.xsemantics.workspace/Xsemantics.setup.

By the way, Xsemantics setup is part of the official Oomph catalog, so you can try it yourself (it’s in the “Github projects” node).

This blog post assumes that you’re already familiar with Oomph and its authoring system.

The initial setup file can be created with the Oomph wizard, so I won’t talk about that.

Source folders in the repository

I found that it is better if all the source folders, including the source folders containing generated code, to be in the git repository. By “source folder” I mean a folder in an Eclipse project which is in the build path as a source folder. Thus, src-gen and xtend-gen should be in the git repository, but NOT their contents (at least, that’s what I want). Remember that git does not store empty folders, so you need to put a .gitignore in such folders stating to ignore everything but itself:

This way, when the containing projects will be imported in Eclipse you won’t risk the Java compiler to stop immediately because of a missing source folder.

Note that this does not seem to always be required: there are projects that can be built anyway, but I found it easier to always include them all.

If you put the .gitignore in more than one *-gen folder you’ll get a warning from Eclipse since it tries to copy those files to the bin folder and it would end up with duplicates. You can avoid this warning by setting the preference “Java Compiler” => “Building” => “Output folder” => “Filtered resources” as shown in the screenshot (I also avoid copying other files into the bin folder):


Use platform URI in MWE2

You should change the grammarURI in your .mwe2 files: they should be platform URIs as opposed to classpath URIs. Otherwise, the MWE2 workflows will fail to find the Xtext grammars when run during the Oomph setup. An example is shown in the following screenshot


Creating a “root” feature for Targlets task

This is not strictly related to Xtext. For the targlets task, in order to specify my own features and bundles, I prefer to specify one single feature which acts as a root for all my Eclipse projects that must be imported in the workspace and that participate to the targ(l)et platform via their requirements. Remember that Oomph will resolve dependencies transitively also for your projects.

To this aim, I define a feature project, e.g., it.xsemantics.workspace (which by the way also contains the Oomph setup file).

In this feature project I specify feature and bundle dependencies to all my other projects (using a feature project just makes the dependency specification easier) in the shape of included plug-ins and included features. Typically the included features are the installable features that you deploy to an update site, and the included plug-ins are the test projects (which are not part of installable features):

oomph-xsemantics3 oomph-xsemantics4

You only need to make sure that transitively these inclusions span all your project’s features and bundles.

However, this won’t help for projects that are neither plug-in projects nor feature projects, like, e.g., all releng projects. Of course you could use the “Project Import” task, but I prefer to create a new “Component Extension” file:oomph-xsemantics5

Here you can specify additional dependencies, in particular, using the type “Project” to refer to Eclipse projects which are not plug-in projects (nor feature projects):


Now, when you define your “Targlets” you can refer to this root feature project, representing all your source projects. Then you can specify additional features for your target platform as usual:


Use variables for Xtext versions

Since I want to have separate Eclipses and workspaces for developing Xsemantics against the current version of Xtext 2.8.4 and the development version 2.9.0 (taken from the nightly update sites), I find it very important to refer to Xtext update sites using Oomph variables (in my case xtext.site and mwe2.size):


The values of such variables are defined in two separate Git branches specifications (you see I have variables also for API baseline settings, but I won’t talk about them since they’re not related to the aim of this post):


I’ll use those variables also for the “P2 director” tasks; this will ensure that the Xtext plug-ins I have in Eclipse will be the same as the ones in the target platform:


Running MWE2

This was the most challenging part: although Oomph provides a “Launch” task, running mwe2 workflows during the workspace setup has always been a problem (at least, that’s what I find in most places on the web).

First of all, you need to run the mwe2 launch AFTER the “Targlets” task and after a “Project Build” task

oomph-xsemantics11 oomph-xsemantics12

For the “Launch” task, you need to use the name of the .launch file, without .launch.

And here’s another small problem: of course the “Project build” task will leave the workspace full of error markers after the execution since the generated Java files are still not there; so the launch of the mwe2 workflow will make the famous popup dialog appear, asking whether you want to cancel the launch because of errors in the workspace… this is very annoying.

To avoid this, you can put a “Preference” task to always disable that dialog (you may want to renable that check later manually, after the workspace is provisioned):

oomph-xsemantics13Now the launch will start automatically without popup dialogs :)

By the way, don’t get fooled by the property name “cancel_launch…”; this actually corresponds to this preference “Continue launch…”:

oomph-xsemantics14Dealing with DSL dependencies

One of the Xsemantics DSL example “FJ cached” extends another DSL example “FJ”, thus, before running the MWE2 for “FJ cached” we must make sure that “FJ” has already been built, i.e., its MWE2 workflow has been executed and its Java sources have been compiled.

So we must insert another “Project Build” task at the right position:


That’s all!

Now the whole setup procedure will run smoothly and at the end all the projects will be imported and will show no sign of error (not even a warning 😉

Other features

This setup also features API baseline setting, and Mylyn Github query.

You may want to try it yourself; as stated above, Xsemantics is part of the official Oomph catalog. The whole procedure might take a few minutes to conclude. During the procedure, as always, you might be asked a few passwords, depending on the choices you made before starting the setup.


Oomph is great great great! :) Ed Merks and Eike Stepper really made a wonderful project :)

I now started to port all my Xtext projects to Oomph. By the way, if your Xtext project is simpler (i.e., no DSL dependencies) you may want to have a look at another example, Java–, which is also part of the official Oomph catalog.

Happy Oomphing! 😉


Be Sociable, Share!

Dealing with Technical Debt with Sonarqube: a case study with Xsemantics

I recently started to play with Sonarqube to reduce “technical debt” and hopefully improve code quality. I’d like to report on my experiences about using Sonarqube to analyze Xsemantics, a DSL for writing rule systems (e.g., type systems) for Xtext languages.

I was already using the Jenkins Continuous Integration server, and while building I was already using Findbugs and Jacoco, thus, I was already analyzing such software, but Sonarqube brings new analysis rules for Java programs and it also integrates results from Findbugs and Jacoco, aggregating all the code quality results in a web site.

In spite of the Jenkins builds Sonarqube detected some issues when I started

xsemantics sonarqube 1

First of all, I had to exclude the src-gen and emf-gen directories (the former is where Xtext generates all its artifacts, and the latter is where Xcore generates the EMF model files); since these are generated files and I did not want to make them part of the analysis. I’ve done such exclusion with a property in the main pom.xml (for readability I split it into lines):

Note that for the moment I’m also excluding tests from the analysis… it is considered best practice to analyse tests as well (and I have many of them), but I wanted to concentrate on the code first. I also excluded other Java files for which issues are reported, like the Xtext Guice modules, due to the wildcards in the method signatures… I have to live with them anyway :)

After that the number of issues reduced a little bit, but there were still some issues to fix; most of them were easy, basically due to Java conventions I hadn’t use (e.g., name of fields and methods or even names of type parameters).

One of the major ones was due to the wrong implementation of the clone method (“super.clone() should be called when overriding Object.clone()” (https://github.com/LorenzoBettini/xsemantics/issues/34).

Another thing that I had never considered was dependency cycles among Java packages and files. Sonarqube reports them. Luckily there were only few of them in Xsemantics, and the hardest part was to read the Dependency Structure Matrix, but in the end I managed to remove them (there must be nothing in the upper triangle to have no cycle):

xsemantics sonarqube 2

To solve the cycles I had to change something in the runtime API (http://xsemantics.sourceforge.net/snapshots-for-xsemantics-1-6-for-xtext-2-7/) but it was basically a matter of moving Java classes into different packages.

Then came the last major issue: Duplicated Code!!! All by itself this issue was estimated with 13 days of technical debt! And most of the duplicated code was in the model inferrer (a concept from Xbase). Moreover, such inferrer is written in Xtend, a cleaner Java, and the Xtend compiler then generates Java code. Thus, Sonarqube analyses the generated Java code, and the detected duplicate code blocks are on the Java code. This means that it takes some time to understand the corresponding original Xtend code. That’s not impossible since Xtend generates clean Java code, but it surely adds some work :)

Before starting to remove duplicated code (around 80 blocks in the generated Java code) the Xtend inferrer was around 1090 lines long (many parts are related to string templates for code generation) corresponding to around 2500 lines of generated Java code! After the refactoring the Xtend inferrer was around 1045 lines long, and the generated Java code reduced to around 2000 lines.

That explains also the reduction of lines of code and complexity:

xsemantics sonarqube 3

But now technical debt is 0 :)

xsemantics sonarqube 4

And it’s nice to look at this dashboard :)

xsemantics sonarqube 5

By the way, I also had to disable some issues I did not agree on (tabulation characters) and avoid reported issues on method name conventions on a specific file (because methods that start with the underline characters _ have a specific meaning in Xtext/Xtend). Instead of disabling them on the Sonarqube web interface, I preferred to disable them using properties in the pom file so that it works across different Sonarqube installations (e.g., I also have a local Sonarqube instance on my machine to do some quick experiments). Such multi properties are not officially supported in the Sonar invocation (e.g., through the sonar runner or via Maven), but I found a workaround: http://stackoverflow.com/questions/21825469/configure-sonar-sonar-issue-ignore-multicriteria-through-maven (but, be careful, it is considered a hack as reported in the mailing list: http://sonarqube.15.x6.nabble.com/sonar-issue-ignore-multicriteria-td5021722.html):

That’s all! I strongly suggest to give Sonarqube a try! :)

Be Sociable, Share!

Switching to Xcore in your Xtext language

This is a followup of my previous post, Switching from an inferred Ecore model to an imported one in your Xtext grammar. The rationale for switching to manually maintained metamodel can be found in the previous post. In this post, instead of using an Ecore file, we will use Xcore,

Xcore is an extended concrete syntax for Ecore that, in combination with Xbase, transforms it into a fully fledged programming language with high quality tools reminiscent of the Java Development Tools. You can use it not only to specify the structure of your model, but also the behavior of your operations and derived features as well as the conversion logic of your data types. It eliminates the dividing line between modeling and programming, combining the advantages of each.

I took inspiration from Jan Köhnlein’s blog post; after switching to a manually maintained Ecore in Xsemantics, I felt the need to further switch to Xcore, since I had started to write many operation implementations in the metamodel, and while you can do that in Ecore, using Xcore is much easier :) Thus in my case I was starting from an existing language, not to mention the use of Xbase (not covered in Jan’s post). Things were not easy, but once the procedure works, it is easily reproducible, and I’ll detail this for a smaller example.

So first of all, let’s create an Xtext project, org.xtext.example.hellocustomxcore, (you can find the sources of this example online at https://github.com/LorenzoBettini/Xtext2-experiments); the grammar of the DSL is not important: this is just an example. We will first start developing the DSL using the automatic Ecore model inference and later we will switch to Xcore.

(the language is basically the same of the previous post).

The grammar of this example is as follows:

and we run the MWE2 generator.

To have something working, we also write an inferrer

With this DSL we can write programs of the shape (nothing interesting, this is just an example)

Now, let’s say we want to check in the validator that there are no elements with the same name; since both “Hello” and “Greeting” have the feature name, we can introduce in the metamodel a common interface with the method getName(). OK, we could achieve this also by introducing a fake rule in the Xtext grammar, but let’s do that with Xcore.

Switching to Xcore

Of course, first of all, you need to install Xcore in your Eclipse.

Before we use the export wizard, we must make sure we can open the generated .genmodel with the “EMF Generator” editor (otherwise the export will fail). If you get an error opening such editor about resolving proxy to JavaJVMTypes.ecore like in the following screenshot…


..then we must tweak the generated .genmodel and add a reference to JavaVMTypes.genmodel: open HelloXcore.genmodel with the text editor, and search for the part (only the relevant part of the line is shown)

and add the reference to the JavaVMTypes.genmodel:

Since we’re editing the .genmodel file, we also take the chance to modify the output folder for the model files to emf-gen (see also later in this section for adding emf-gen as a source folder):

And we remove the properties that relate to the edit and the editor plug-ins (since we don’t want to generate them anyway):

Now save the edited file, refresh the file in the workspace by selecting it and pressing F5 (yes, also this operation seems to be necessary), and this time you should be able to open it with the “EMF Generator” editor. We can go on exporting the Xcore file.

We want the files generated by Xcore to be put into the emf-gen source folder; so we add a new source folder to our project, say emf-gen, where all the EMF classes will be generated; we also make sure to include such folder in the build.properties file.

First, we create an .xcore file starting from the generated .genmodel file:

  • navigate to the HelloXcore.genmodel file (it is in the directory model/generated)
  • right click on it and select “Export Model…”
  • in the dialog select “Xcore”
  • The next page should already present you with the right directory URI
  • In the next page select the package corresponding to our DSL, org.xtext.example.helloxcore.helloxcore (and choose the file name for the exported .xcore file corresponding Helloxcore.xcore file)
  • Then press Finish
  • If you get an error about a missing EObjectDescription, remove the generated (empty) Helloxcore.xcore file, and just repeat the Export procedure from the start, and the second time it should hopefully work


The second time, the procedure should terminate successfully with the following result:

  • The xcore file, Helloxcore.xcore has been generated in the same directory of the .genmodel file (and the xcore file is also opened in the Xcore editor)
  • A dependency on org.eclipse.emf.ecore.xcore.lib has been added to the MANIFEST.MF
  • The new source folder emf-gen is full of compilation errors


Remember that the model files will be automatically generated when you modify the .xcore file (one of the nice things of Xcore is indeed the automatic building).

Fixing the Compilation Errors

These compilation errors are expected since Java files for the model are both in the src-gen and in the emf-gen folder. So let’s remove the ones in the src-gen folders (we simply delete the corresponding packages):


After that, everything compile fines!

Now, you can move the Helloxcore.xcore file in the “model” directory, and remove the “model/generated” directory.

Modifying the mwe2 workflow

In the Xtext grammar, HelloXcore.xtext, we replace the generate statement with an import:

The DirectoryCleaner fragment related the “model” directory should be removed (otherwise it will remove our Helloxcore.xcore file as well); and we don’t need it anymore after we manually removed the generated folder with the generated .ecore and .genmodel files.

Then, in the language part, you need to loadResource the XcoreLang.xcore, the Xbase and Ecore ecore and genmodel, and finally the xcore file you have just exported, Helloxcore.xcore.

We can comment the ecore.EMFGeneratorFragment (since we manually maintain the metamodel from now on).

The MWE2 files is now as follows (I highlighted the modifications):

Before running the workflow, you also need to add org.eclipse.emf.ecore.xcore as a dependency in your MANIFEST.MF.

We can now run the mwe2 workflow, which should terminate successfully.

We must now modify the plugin.xml (note that there’s no plugin.xml_gen anymore), so that the org.eclipse.emf.ecore.generated_package extension point contains the reference to the our Xcore file:

Fixing Junit test problems

As we saw in the previous post, Junit tests do not work anymore with errors of the shape

All we need to do is to modify the StandaloneSetup in the src folder (NOT the generated one, since it will be overwritten by subsequent MWE2 workflow runs) and override the register method so that it performs the registration of the EPackage (as it used to do before):

And now the Junit tests will run again.

Modifying the metamodel with Xcore

We can now customize our metamodel, using the Xcore editor.

For example, we add the interface Element, with the method getName() and we make both Hello and Greeting implement this interface (they both have getName() thus the implementation of the interface is automatic).

Using the Xcore editor is easy, and you have content assist; as soon as you press save, the Java files will be automatically regenerated:


We also add a method getElements() to the Model class returning an Iterable<Element>(containing both the Hello and the Greeting objects). This time, with Xcore, it is really easy to do so (compare that with the procedure of the previous post, requiring the use of EAnnotation in the Ecore file), since Xcore uses Xbase expression syntax for defining the body of the operations (with full content assist, not to mention automatic import statement insertions). See also the generated Java code on the right:


And now we can implement the validator method checking duplicates, using the new getElements() method and the fact that now both Hello and Greeting implement Element:

That’s all! I hope you found this tutorial useful :)


Be Sociable, Share!

Switching from an inferred Ecore model to an imported one in your Xtext grammar

When you use Xtext for developing your language the Ecore model for the AST is automatically derived/inferred from the grammar. If your DSL is simple, this automatic meta-model inference is usually enough. However, there might be cases where you need more control on the meta-model and in such cases you will want to switch from an inferred Ecore model to a an imported one, which you will manually maintain. This is documented in the Xtext documentation, and in some blog posts. When I needed to switch to an imported Ecore model for Xsemantics, things have not been that easy, so I thought to document the steps to perform some switching in this tutorial, using a simple example. (I should have talked about that in my Xtext book, but at that time I ran out of pages so there was no space left for this subject :)

So first of all, let’s create an Xtext project, org.xtext.example.hellocustomecore, (you can find the sources of this example online at https://github.com/LorenzoBettini/Xtext2-experiments); the grammar of the DSL is not important: this is just an example. We will first start developing the DSL using the automatic Ecore model inference and later we will switch to an imported Ecore.

The grammar of this example is as follows (to make things more interesting, we will also use Xbase):

and we run the MWE2 generator.

To have something working, we also write an inferrer

With this DSL we can write programs of the shape (nothing interesting, this is just an example)

Now, let’s say we want to check in the validator that there are no elements with the same name; since both “Hello” and “Greeting” have the feature name, we can introduce in the Ecore model a common interface with the method getName(). OK, we could achieve this also by introducing a fake rule in the Xtext grammar, but let’s switch to an imported Ecore model so that we can manually modify that.

Switching to an imported Ecore model

First of all, we add a new source folder to our project (you must create it with File -> New -> Source Folder, or if you create it as a normal folder, you then must add it as a source folder with Project -> Properties -> Lava Build Path: Source tab), say emf-gen, where all the EMF classes will be generated; we also make sure to include such folder in the build.properties file:

Remember that, at the moment, the EMF classes are generated into the src-gen folder, together with other Xtext artifacts (e.g., the ANTLR parser):


Xtext generates the inferred Ecore model file and the GenModel file into the folder model/generated


This is the new behavior introduced in Xtext 2.4.3 by the fragment ecore.EMFGeneratorFragment that replaces the now deprecated ecore.EcoreGeneratorFragment; if you still have the deprecated fragment in your MWE2 files, then the Ecore and the GenModel are generated in the src-gen folder.

Let’s rename the “generated” folder into “custom” (if in the future for any reason we want to re-enable Xtext Ecore inference, our custom files will not be overwritten):


NOTE: if you simply move the .ecore and .genmodel file into the directory model, you will not be able to open the .ecore file with the Ecore editor: this is due to the fact that this Ecore file refers to Xbase Ecore models with a relative path; in that case you need to manually adjust such references by opening the .ecore file with the text editor.

From now on, remember, we will manually manage the Ecore file.

Now we change the GenModel file, so that the EMF model classes are generated into emf-gen instead of src-gen:

imported-ecore-genmodelWe need to change the MWE2 file as follows:

  • Enable the org.eclipse.emf.mwe2.ecore.EcoreGenerator fragment that will generate the EMF classes using our custom Ecore file and GenModel file; indeed, you must refer to the custom GenModel file; before that we also run the DirectoryCleaner on the emf-gen folder (this way, each time the EMF classes are generated, the previous classes are wiped out); enable these two parts right after the StandaloneSetup section;
  • Comment or remove the DirectoryCleaner element for the model directory (otherwise the workflow will remove our custom Ecore and GenModel files);
  • In the language section we load our custom Ecore file,
  • and we disable ecore.EMFGeneratorFragment (we don’t need that anymore, since we don’t want the Ecore model inference)

The MWE2 files is now as follows (I highlighted the modifications):

We add the dependency org.eclipse.xtext.ecore in the MANIFEST.MF:imported-ecore-manifestIn the Xtext grammar we replace the generate statement with an import statement:

Now we’re ready to run the MWE2 workflow, and you should get no error (if you followed all the above instructions); you can see that now the EMF model classes are generated into the emf-gen folder (the corresponding packages in the src-gen folders are now empty and you can remove them):


We must now modify the plugin.xml (note that there’s no plugin.xml_gen anymore), so that the org.eclipse.emf.ecore.generated_package extension point contains the reference to the new GenModel file:


If you try the editor for the DSL it will still work; however, the Junit tests will fail with errors of this shape:

That’s because the generated StandaloneSetup does not register the EPackage anymore, see the diff:


All we need to do is to modify the StandaloneSetup in the src folder (NOT the generated one, since it will be overwritten by subsequent MWE2 workflow runs) and override the register method so that it performs the registration of the EPackage:

And now the Junit tests will run again.

Modifying the Ecore model

We can now customize our Ecore model, using the Ecore editor and the Properties view.

For example, we add the interface Element, with the method getName() and we make both Hello and Greeting implement this interface (they both have getName() thus the implementation of the interface is automatic).

imported-ecore-custom-ecore1 imported-ecore-custom-ecore2 imported-ecore-custom-ecore3

We also add a method getElements() to the Model class returning an Iterable<Element> (containing both the Hello and the Greeting objects)


and we implement that method using an EAnnotation, using the source “http://www.eclipse.org/emf/2002/GenModel” and providing a body

imported-ecore-custom-ecore5 imported-ecore-custom-ecore6

With the following implementation

Let’s run the MWE2 workflow so that it will regenerate the EMF classes.

And now we can implement the validator method checking duplicates, using the new getElements() method and the fact that now both Hello and Greeting implement Element:

That’s all! I hope you found this tutorial useful :)

Be Sociable, Share!

Using the Xtend compiler in Buckminster builds

Up to now, I was always putting the Xtend generated Java files in my git repositories (for my Xtext projects), since I still hadn’t succeeded in invoking the Xtend standalone compiler in a Buckminster build. Dennis Hübner published a post with some hints on how to achieve that, but that never worked for me (and apparently it did not work for other users).

After some experiments, it seems I finally managed to trigger Xtend compilation in Buckminster builds, and in this post I’ll show the steps to achieve that (I’m using an example you can find on Github).

The main problems I had to solve were:

  • how to pass the classpath to the Xtend compiler
  • how to deal with chicken-and-egg problems (dependencies among Java and Xtend classes).

IMPORTANT: the build process described here uses a new flag for the Buckminster’s build command, which has been recently added; thus, you must make sure you have an updated version of Buckminster headless (from 4.3 repository).

The steps to perform can be applied to your projects as well; they are simple and easy to reproduce. In this blog post I’ll try to explain them in details.

This blog post assumes that you are already familiar with setting up a Buckminster build.

The example

The example I’m using is an Xtext DSL (just the Greeting example using Xbase), with many .xtend files and with the standard structure:

  • org.xtext.example.hellobuck, the runtime plugin,
  • org.xtext.example.hellobuck.ui, the ui plugin, which uses Xtend classes defined in the runtime plugin,
  • org.xtext.example.hellobuck.tests, the tests plugin, which uses Xtend classes defined in the runtime and in the ui plugin,
  • org.xtext.example.hellobuck.sdk, the SDK feature for the DSL.

Furthermore, we have two additional projects created by the Xtext Buckminster Wizard:

  • org.xtext.example.hellobuck.buckminster, the releng project,
  • org.xtext.example.hellobuck.site, the feature project for creating the p2 repository,

I blogged about the Xtext Buckminster Wizard in the past, and indeed this example is a fork of the example presented in that blog post.

Creating a launch configuration for the Xtend compiler

The first step consists in creating a Java launch configuration in the runtime plugin project that invokes the Xtend standalone compiler. This was shown in Dennis’ original post, but you need to change a few things. Here’s the XtendCompiler.launch file to put in the org.xtext.example.hellobuck runtime plugin project (of course you can call the launch file whateven you want):

This launch configuration can be reused in other projects, provided the highlighted lines are changed accordingly, since they refer to the containing project.

An important part of this launch configuration is the PROGRAM_ARGUMENTS that are passed to the Xtend compiler, in particular the -classpath argument. This was the main problem I experienced in the past (and that I saw in all the other posts in the forum): the Xtend compiler needs to find the Java classes your Xtend files depend upon and thus you need to pass a valid -classpath argument. But we can simply reuse the classpath of the containing project :)

Add dependency for Xtend standalone compiler

This launch configuration calls the Java application org.eclipse.xtend.core.compiler.batch.Main thus you must add a dependency on the corresponding bundle in your MANIFEST.MF. The bundle you need to depend on is org.eclipse.xtend.standalone (the dependency can be optional):



Test the launch in your workbench

You can test this launch configuration from Eclipse, with Run As => Java Application. In the Console view you should see something like:

This will give you confidence that the launch configuration works correctly and that all dependencies for invoking the Xtend compiler are in place.

Add an XtendCompiler.launch in the other projects

You must now add an XtendCompiler.launch in all the other projects containing Xtend files. In our example we must add it to the ui and the tests projects.

You can copy the one you have already created but MAKE SURE you update the relevant 3 parts according to the containing projects! See the highlighted lines above.

NOTE: you do NOT need to add a dependency on org.eclipse.xtend.standalone in the MANIFEST.MF of the ui and tests projects: they depend on the runtime plugin project which already has that dependency.

You may want to run the XtendCompiler.launch also in these projects from the Eclipse workbench, again to get confidence that you configured the launch configurations correctly.

IMPORTANT: when the Xtend compiler compiles the files in the ui and tests project, you will see some ERROR lines, e.g.,

From what I understand, these errors do not prevent the Xtend compiler to successfully generate Java files (see the final INFO line) and the procedure terminates successfully. Thus, you can ignore these errors. If the Xtend compiler really cannot produce Java files it will terminate with a final error.

Configure the headless build

Now it’s time to configure the Buckminster headless build so that it runs the Xtend compiler. We created .launch files because one of the cool things of Buckminster is that it can seamlessly run the launch files.

The tricky part here is that since we perform a clean build, there is a chicken-and-egg scenario

  • no Java files have been compiled,
  • most Java files import Java files created by Xtend
  • the Xtend files import Java classes

To solve these problems we perform an initial clean build; this will run the Java compiler and such compilation will terminate with errors. We expect that, due to the chicken-and-egg situation. However, this will create enough .class files to run the Xtend compiler! It is important to run the build command with the (new) flag –continueonerror, otherwise the whole build will fail.

After running XtendCompiler.launch in the org.xtext.example.hellobuck runtime project, we run another build –continueonerror so that the Java files generated by the Xtend compiler will be compiled by Java. We then proceed similarly for the ui and the tests project:

Then, your build can proceed as usual (at this point I prefer to perform a clean build): run the tests (both plain Junit and Plug-in Junit tests) and create the p2 repository:

The complete commands file can be seen here.

Executing the headless build

You can now run your headless build on your machine or on Jenkins.

My favorite way of doing that is by using an ANT script. The whole ANT script can be found in the example.

This script also automatically installs Buckminster headless if not present.

Before executing the commands file, it also removes the contents of the xtend-gen folder in all the projects; this way you are sure that no stale generated Java files are there.

Of course, you should now remove the xtend-gen folder from your git repository (and put it in the .gitignore file).

In Jenkins you can configure an Invoke Ant build step as shown in the screenshot (“Start Xvfb before the build, and shut it down after” is required to execute Plug-in Junit tests; we also pass an option to install Buckminster headless in the job’s workspace).


Try the example

You can just clone the git repository, and then

As noted above, this will also install Buckminster headless if not found in the location specified by the property buckminster.home. This script will take some time, especially the first time, since it will materialize the target platform.

Hope you find this blog post useful! :)

Be Sociable, Share!

The book on Xtext is out

My book on Xtext, “Implementing Domain-Specific Languages with Xtext and Xtend” is now available on Packt website! Get it while it’s hot! :)

You can find the outline and an example chapter at


Many thanks to the reviewers of the book: Jan Koehnlein, Henrik Lindberg, Pedro J. Molina, and Sebastian Zarnekow!

The sources of the examples presented in the book are available at https://github.com/LorenzoBettini/packtpub-xtext-book-examples

0304OS_mockupcover_normalI would also like to thank all the people from Packt I dealt with.


Be Sociable, Share!

The XImportSection in Xbase 2.4

I know that Xtext 2.4 has not been released yet, but I could not resist blogging about a very cool new feature in Xbase: improved automatic import functionalities!

Actually, import functionalities were already great when using Xbase also in previous versions of Xtext, but now they provide a much better experience for the user of your DSL! Indeed, all the import functionalities you are used to with JDT (like automatic import insertion, and organize imports) are available also for your Xbase language; these features were already available in Xtend, and they have been ported to Xbase itself.

At the time of writing, you need to get the very latest updates of Xtext 2.4, using the update site http://download.eclipse.org/modeling/tmf/xtext/updates/composite/latest/ .

Before you used to do something like

Now, you can use in your grammar the new Xbase rule: XImportSection:

In this post I’m reusing some experiments you can find here (https://github.com/LorenzoBettini/Xtext2-experiments, I had blogged about these experiments in previous posts).

If you now rerun the MWE2 generator, and make sure you merge the plugin.xml_gen with plugin.xml in the .ui project, your editor will provide some interesting features for free (if you use my examples, you can find a project wizard “New Project” => “Xtext” => “HelloInferrer Project”):

Imports with wildcards are deprecated:


You now have the context menu “Organize Imports” (Shift + Control + O); try that one in the presence of such deprecation warning and imports are organized for you:


Similarly, unused imports are reported as warnings:


Again, use “Organize Imports” to fix that!

The new feature I like most is the automatic insertion of imports! (just like in JDT and Xtend): try to get content assist for a Java type, for instance,


Accept a proposal and the import will be automatically inserted (instead of the fully qualified name):


Xtext rocks! :)

Be Sociable, Share!

Building Xtext projects with Buckminster

One of the new cool features which came with Xtext 2.3.0 is a wizard which generates all the artifacts to build your Xtext DSL project with Buckminster; this will allow you to easily build your Xtext project p2 site and also to build your project and its tests headlessly (e.g., from the command line, and, more importantly from within a continuous integration system like Jenkins).

This new feature was not advertised much, nor documented, so I decided to write a tutorial about that, also with the permission of the developer of this feature Dennis Hübner.

In this tutorial I will show how to

  • create the p2 repository from the IDE
  • build your project (and run the tests) headlessly from the command line (through ant)
  • build your project (and run the tests) headlessly in a continuous integration system like Jenkins

In the end, I will also try to describe/explain all the Buckminster files that the wizard created for you, so that one can customize them.

The sources used in this tutorial can be found at


Create an Xtext project

So, first of all, let’s create a simple Xtext hello project; for this example I will use the following settings:

  • project name: org.xtext.example.hellobuck
  • name: org.xtext.example.hellobuck.HelloBuck
  • extension: greetings
  • check “Create SDK feature project”

and then, of course, we generate Xtext artifacts.

The DSL is basically the standard Greetings DSL; in this case, I’m using the JvmModelInferrer to generate Java classes from “Hello” specifications. Indeed, the DSL itself is not important in this tutorial. I’ve also added two Junit tests in the corresponding org.xtext.example.hellobuck.tests project:

Note the org.xtext.example.hellobuck.tests.launch Launch configuration that Xtext created for you (which basically runs all the Junit tests in this project. We will see how this will be used later.

Use the Xtext Build with Buckminster Wizard

Now we can use the Xtext wizard “Build with Buckminster”: New => Other => Xtext => Continuous Integration => Build with Buckminster:

You will have to specify the sdk feature of your Xtext project (that has been created by the Xtext new project wizard); the releng and site projects will have predefined names. For the Buckminster installation directory you have 3 choices:

  • if you have already an headless installation of Buckminster in your system you can specify the path;
  • you can install it using the link in the dialog
  • you can leave that path empty and specify it later in the build.ant generated file (or pass it on the command line when invoking ant)

In any case, specifying the headless Buckminster installation path is only useful if you intend to build your projects headlessly with ant (it is not requested for building in the IDE, nor if you plan to build it with Jenkins by setting job using the Buckminster Jenkins plugin, as we will see later).

Before pressing Finish, we also want to specify the tests to launch during the headless build, so we press the button Add and we select the launch configuration in the .tests project which the Xtext wizard created for us.

If you want to build an existing Xtext project and you do not have that launch configuration, all you need to do is to simply create a new plain Junit launch configuration from the IDE, for instance, to run your existing Junit test suite, and save it in the .tests project yourself. You can also configure the headless build to run tests later on.

When the wizard finishes, you will end up with two additional projects

  • org.xtext.example.hellobuck.buckminster with all the Buckminster (CQUERY and RMAP, build.ant, etc.) files to build your project headlessly;
  • org.xtext.example.hellobuck.site which is a feature project to create the p2 repository for your Xtext project.

Build the p2 repository from the IDE

Although the .buckminster project created by the wizard is thought to be used in an headless environment, the .site project is useful also to build the p2 repository from the IDE itself! Indeed, if you created an old style update site, you could build the update site from the IDE itself. But if you switched to the new category.xml format, then building the corresponding p2 repository from the IDE is not straightforward.

If you want to run Buckminster from Eclipse, you first need to install it in the IDE of course, by using this repository: http://download.eclipse.org/tools/buckminster/updates-4.2

Before creating the p2 repository, you may want to tweak some Buckminster properties, for instance, you may want to create a buckminster.properties file in the .buckminster project (or in the .site project) like the following

so that Buckminster will generate all its artifacts in the tmp/hellobuck directory of your home folder.

We now right click on the .site project, and select Buckminster => Invoke Action… , we select site.p2 action, and optionally refer to the buckminster.properties file

When the action finishes, you will file the p2.site in the directory $HOME/tmp/hellobuck/build/org.xtext.example.hellobuck.site_1.0.0-eclipse.feature/site.p2/

Build with ant

The wizard created for us, in the .buckminster project a build.ant file that we can use to build the project, run the tests, and create the p2 repository with ant

Note that this file has the location of your Buckminster headless installation path hardcoded (recall the wizard we used before for creating Buckminster projects); if that is not correct, you can still pass this information to ant when invoking it

This ant script will build your project in the directory buildroot which will be created in the directory where all your projects are located (e.g., in the workspace or in another path); in particular it will

  1. materialize the target platform for your project (the first time you run it), and this requires an Internet connection and might take some time
  2. build your projects
  3. run the Junit tests
  4. build the p2 site repository

This should be the output (where WORKSPACE is actually the location of your projects):

It will also tell you where you can find the generated p2 site.

IMPORTANT: there’s a bug in the currently generated build.ant file, which prevent you from building with ant in Windows; until the fixed version of the wizard is released, you need to make sure that the build.ant has the right contents, as in this example source.

Build with Jenkins

To build your Xtext project in Jenkins you can either create a Job which uses ant and the build.ant script, or create a Job which uses the Buckminster Jenkins plugin (I’ve also blogged about that). I will detail both ways. Both jobs have in common the access to the git repository

A Job using ant

This requires that you configured an ant installation in your Jenkins, and that you already have a working version of Buckminster headless in Jenkins; you need to specify them in the ant build step configuration (the screeshots, of course, refer to the paths and values for my Jenkins installation):

then, we configure post build actions to archive both the artifacts (the p2 repository) and the Junit results:

A job using Buckminster plugin

To configure a Buckminster build step, we prepare a text file in the .buckminster projects with the Buckminster commands to execute (we take inspiration from the commands.txt file that the wizard created), we call it jenkins-commands.txt:

Then we configure a build step (this relies on a Buckminster installation that we have already configured in Jenkins), note that we refer to the jenkins-commands.txt we created above (an alternative would be to copy the commands directly in the text area “Commands”):

To configure the post build actions, since we used different output paths, we have to specify them accordingly (in particular, we have not used the output directory buildroot, thus the default will be used):

Details and Customization

It may be interesting to learn more about the files that the Xtext Buckminster wizard generated for you (at least, I personally found instructive to look at them, and learned something about Buckminster :) most of which I described in another post of this blog). The details might also help you if you need to customize the generated files.

The .buckminster project contains a Component Specification (CSPEC), buckminster.cspec, which looks like

thus, due to its dependencies, it represents the target platform for your Xtext project. If your Xtext project required some further specific dependencies to build and test, this is the place to express them!

For instance, in an Xtext project of mine, Xsemantics, I run also some SwtBot tests, which also rely on pde, thus, the dependencies in the corresponding .buckminster project’s buckminster.cspec look like

To materialize components and dependencies with Buckminster, you need a Resource Map (RMAP), and the wizard created two maps: one containing p2 repositories location (for materializing the features of your target platform), called projects-platform.rmap and one for binding your projects to the workspace (when building headlessly), called project.rmap.

projects-platform.rmap looks like

This basically tells Buckminster to take all Xtext stuff (including additional components like Google Guice and Antlr) from the Xtext main repository, and to use the main Eclipse Juno release repository for everything else. However, it also redirects to project.rmap for resolving the main Buckminster component of our project org.xtext.example.hellobuck.buckminster.

Again, if your target platform needs additional features from different sites, this is the place where to express them; reusing the example above which also needed SwtBot the rmap would also have these additional stuff:

The project.rmap tells Buckminster where to find the projects of our Xtext project (so that it can bind it to the workspace when building headlessly):

This rmap assumes that all the Eclipse projects of your Xtext project are in the same base directory; if this is not the case, because you split them, for instance, in plugins, features, doc, etc. directories, then you need to tweak this rmap accordingly, for instance, again, taken from Xsemantics,

Then, you have to Component Query files (CQUERY): projects-platform.cquery, to materialize the target platform, and project.cquery, to materialize your projects in the workspace. The first one could also be used in the IDE to actually materialize the target platform in your workspace (instead of using a target definition file); the second one is useful when building headless.


Note that this cquery has an advisor node to skip .source components: thus, the materialized target platform will not contain the sources for all the features. This is useful headlessly to reduce the time to materialize the target platform. But if you use it also for materializing the target platform in the IDE, then sources are useful to inspect Java classes, and you might want to remove this advisor node (or simply create another cquery only to be used in the IDE, without that advisor node). This query refers to projects-platform.rmap.


As it usually happens when using Buckminster, the cquery for materializing the projects in the (headless) workspace refers to a single feature project which, transitively refers to all the bundles and additional features of your application. Usually, this is the feature project for building the p2 repository, like in this case.

“But wait! the site feature project does not refer to test projects, since I do not want to include them in my p2 repository! So, how can Buckminster run my tests headlessly if they are not materialized?”

If you look at the feature project .site, you will note that it contains a Component Extension (CSPEX) which allows to extend the automatically inferred Component Specification for the feature project with additional specifications. In this case the buckminster.cspex contains

You see that this specification adds a dependency to the org.xtext.example.hellobuck.tests project, so that, when Buckminster materializes org.xtext.example.hellobuck.site, and its dependencies, it will also materialize org.xtext.example.hellobuck.tests. You can use this technique to materialize additional projects; for instance, in from Xsemantics, I have a main feature for tests, and a tests project for each example, thus buckminster.cspex reads as follows

Finally, in the .site project, you will also find the feature.xml

which includes the .sdk feature of your project (Remember: when building the site.p2 on a feature project with Buckminster, you will build a p2 repository NOT for the very feature, but for the included features), and the category.xml to give categories to your features in the p2 repository:

Note that the .sdk.source feature is not a “real” feature project in your workspace, but, by default, Buckminster will automatically build a source jar for all your features and bundles.

That’s all! Many thanks to the author of this wizard, Dennis Hübner, for making it and for many suggestions!

Have fun with Bucky and Xtext :)


Be Sociable, Share!