Tag Archives: xtext

Switching to Xcore in your Xtext language

This is a followup of my previous post, Switching from an inferred Ecore model to an imported one in your Xtext grammar. The rationale for switching to manually maintained metamodel can be found in the previous post. In this post, instead of using an Ecore file, we will use Xcore,

Xcore is an extended concrete syntax for Ecore that, in combination with Xbase, transforms it into a fully fledged programming language with high quality tools reminiscent of the Java Development Tools. You can use it not only to specify the structure of your model, but also the behavior of your operations and derived features as well as the conversion logic of your data types. It eliminates the dividing line between modeling and programming, combining the advantages of each.

I took inspiration from Jan Köhnlein’s blog post; after switching to a manually maintained Ecore in Xsemantics, I felt the need to further switch to Xcore, since I had started to write many operation implementations in the metamodel, and while you can do that in Ecore, using Xcore is much easier :) Thus in my case I was starting from an existing language, not to mention the use of Xbase (not covered in Jan’s post). Things were not easy, but once the procedure works, it is easily reproducible, and I’ll detail this for a smaller example.

So first of all, let’s create an Xtext project, org.xtext.example.hellocustomxcore, (you can find the sources of this example online at https://github.com/LorenzoBettini/Xtext2-experiments); the grammar of the DSL is not important: this is just an example. We will first start developing the DSL using the automatic Ecore model inference and later we will switch to Xcore.

(the language is basically the same of the previous post).

The grammar of this example is as follows:

grammar org.xtext.example.helloxcore.HelloXcore with 
	org.eclipse.xtext.xbase.Xbase

generate helloxcore "http://www.xtext.org/example/hellocustomxcore/HelloXcore"

Model:
    importSection=XImportSection?
    hellos+=Hello*
    greetings+=Greeting*
;

Hello:
    'Hello' name=ID '!'
;

Greeting:
    'Greeting' name=ID expression = XExpression
;

and we run the MWE2 generator.

To have something working, we also write an inferrer

def dispatch void infer(Model element, IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
   		acceptor.accept(element.toClass("generated." + element.eResource.URI.lastSegment.split("\\.").head.toFirstUpper))
   			.initializeLater([
   				for (hello: element.hellos) {
   					members += hello.toMethod
   							("say" + hello.name.toFirstUpper, hello.newTypeRef(String)) [
   						body = '''return "Hello «hello.name»";'''
   					]
   				}
   				for (greeting : element.greetings) {
   					members += greeting.toMethod
   							("say" + greeting.name.toFirstUpper, greeting.newTypeRef(String)) [
   						body = greeting.expression
   					]
   				}
   			])
   	}

With this DSL we can write programs of the shape (nothing interesting, this is just an example)

Hello foo!

Greeting bar {
    sayFoo() + "bar"
}

Now, let’s say we want to check in the validator that there are no elements with the same name; since both “Hello” and “Greeting” have the feature name, we can introduce in the metamodel a common interface with the method getName(). OK, we could achieve this also by introducing a fake rule in the Xtext grammar, but let’s do that with Xcore.

Switching to Xcore

Of course, first of all, you need to install Xcore in your Eclipse.

Before we use the export wizard, we must make sure we can open the generated .genmodel with the “EMF Generator” editor (otherwise the export will fail). If you get an error opening such editor about resolving proxy to JavaJVMTypes.ecore like in the following screenshot…

gemodel_problems

..then we must tweak the generated .genmodel and add a reference to JavaVMTypes.genmodel: open HelloXcore.genmodel with the text editor, and search for the part (only the relevant part of the line is shown)

usedGenPackages="../../../org.eclipse.xtext.xbase/model/Xbase.genmodel#//xbase

and add the reference to the JavaVMTypes.genmodel:

usedGenPackages="../../../org.eclipse.xtext.common.types/model/JavaVMTypes.genmodel#//types ../../../org.eclipse.xtext.xbase/model/Xbase.genmodel#//xbase

Since we’re editing the .genmodel file, we also take the chance to modify the output folder for the model files to emf-gen (see also later in this section for adding emf-gen as a source folder):

modelDirectory="/org.xtext.example.helloxcore/emf-gen"

And we remove the properties that relate to the edit and the editor plug-ins (since we don’t want to generate them anyway):

editDirectory="/org.xtext.example.helloxcore.edit/src"
editorDirectory="/org.xtext.example.helloxcore.editor/src"
editPluginID="org.xtext.example.helloxcore.edit"
editorPluginID="org.xtext.example.helloxcore.editor"

Now save the edited file, refresh the file in the workspace by selecting it and pressing F5 (yes, also this operation seems to be necessary), and this time you should be able to open it with the “EMF Generator” editor. We can go on exporting the Xcore file.

We want the files generated by Xcore to be put into the emf-gen source folder; so we add a new source folder to our project, say emf-gen, where all the EMF classes will be generated; we also make sure to include such folder in the build.properties file.

First, we create an .xcore file starting from the generated .genmodel file:

  • navigate to the HelloXcore.genmodel file (it is in the directory model/generated)
  • right click on it and select “Export Model…”
  • in the dialog select “Xcore”
    gemodel_export1
  • The next page should already present you with the right directory URI
    gemodel_export2
  • In the next page select the package corresponding to our DSL, org.xtext.example.helloxcore.helloxcore (and choose the file name for the exported .xcore file corresponding Helloxcore.xcore file)
    gemodel_export3
  • Then press Finish
  • If you get an error about a missing EObjectDescription, remove the generated (empty) Helloxcore.xcore file, and just repeat the Export procedure from the start, and the second time it should hopefully work

gemodel_export4

The second time, the procedure should terminate successfully with the following result:

  • The xcore file, Helloxcore.xcore has been generated in the same directory of the .genmodel file (and the xcore file is also opened in the Xcore editor)
  • A dependency on org.eclipse.emf.ecore.xcore.lib has been added to the MANIFEST.MF
  • The new source folder emf-gen is full of compilation errors

gemodel_export5

Remember that the model files will be automatically generated when you modify the .xcore file (one of the nice things of Xcore is indeed the automatic building).

Fixing the Compilation Errors

These compilation errors are expected since Java files for the model are both in the src-gen and in the emf-gen folder. So let’s remove the ones in the src-gen folders (we simply delete the corresponding packages):

gemodel_export6

After that, everything compile fines!

Now, you can move the Helloxcore.xcore file in the “model” directory, and remove the “model/generated” directory.

Modifying the mwe2 workflow

In the Xtext grammar, HelloXcore.xtext, we replace the generate statement with an import:

//generate helloxcore "http://www.xtext.org/example/helloxcore/HelloXcore"
import "http://www.xtext.org/example/helloxcore/HelloXcore"

The DirectoryCleaner fragment related the “model” directory should be removed (otherwise it will remove our Helloxcore.xcore file as well); and we don’t need it anymore after we manually removed the generated folder with the generated .ecore and .genmodel files.

Then, in the language part, you need to loadResource the XcoreLang.xcore, the Xbase and Ecore ecore and genmodel, and finally the xcore file you have just exported, Helloxcore.xcore.

We can comment the ecore.EMFGeneratorFragment (since we manually maintain the metamodel from now on).

The MWE2 files is now as follows (I highlighted the modifications):

...
Workflow {
    bean = StandaloneSetup {
    	scanClassPath = true
    	platformUri = "${runtimeProject}/.."
    	// The following two lines can be removed, if Xbase is not used.
    	registerGeneratedEPackage = "org.eclipse.xtext.xbase.XbasePackage"
    	registerGenModelFile = "platform:/resource/org.eclipse.xtext.xbase/model/Xbase.genmodel"
    }

    component = DirectoryCleaner {
    	directory = "${runtimeProject}/src-gen"
    }

      // switch to xcore
//    component = DirectoryCleaner {
//    	directory = "${runtimeProject}/model"
//    }

    component = DirectoryCleaner {
    	directory = "${runtimeProject}.ui/src-gen"
    }

    component = DirectoryCleaner {
    	directory = "${runtimeProject}.tests/src-gen"
    }

    component = Generator {
    	pathRtProject = runtimeProject
    	pathUiProject = "${runtimeProject}.ui"
    	pathTestProject = "${runtimeProject}.tests"
    	projectNameRt = projectName
    	projectNameUi = "${projectName}.ui"
    	encoding = encoding
    	language = auto-inject {
    		// switch to xcore
        	loadedResource = "platform:/resource/org.eclipse.emf.ecore.xcore.lib/model/XcoreLang.xcore"
			loadedResource = "classpath:/model/Xbase.ecore"
    		loadedResource = "classpath:/model/Xbase.genmodel"
    		loadedResource = "classpath:/model/Ecore.ecore"
			loadedResource = "classpath:/model/Ecore.genmodel"
        	loadedResource = "platform:/resource/${projectName}/model/Helloxcore.xcore"

    		uri = grammarURI

    		// Java API to access grammar elements (required by several other fragments)
    		fragment = grammarAccess.GrammarAccessFragment auto-inject {}

    		// generates Java API for the generated EPackages
    		// switch to xcore
    		// fragment = ecore.EMFGeneratorFragment auto-inject {}
...

Before running the workflow, you also need to add org.eclipse.emf.ecore.xcore as a dependency in your MANIFEST.MF.

We can now run the mwe2 workflow, which should terminate successfully.

We must now modify the plugin.xml (note that there’s no plugin.xml_gen anymore), so that the org.eclipse.emf.ecore.generated_package extension point contains the reference to the our Xcore file:

<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.0"?>

<plugin>

  <extension point="org.eclipse.emf.ecore.generated_package">
    <package 
       uri = "http://www.xtext.org/example/helloxcore/HelloXcore" 
       class = "org.xtext.example.helloxcore.helloxcore.HelloxcorePackage"
       genModel = "model/Helloxcore.xcore" /> 

  </extension>

</plugin>

Fixing Junit test problems

As we saw in the previous post, Junit tests do not work anymore with errors of the shape

org.eclipse.xtext.parser.ParseException: java.lang.IllegalStateException: Unresolved proxy http://www.xtext.org/example/helloxcore/HelloXcore#//Hello. Make sure the EPackage has been registered.

All we need to do is to modify the StandaloneSetup in the src folder (NOT the generated one, since it will be overwritten by subsequent MWE2 workflow runs) and override the register method so that it performs the registration of the EPackage (as it used to do before):

public class HelloXcoreStandaloneSetup extends HelloXcoreStandaloneSetupGenerated{

	public static void doSetup() {
		new HelloXcoreStandaloneSetup().createInjectorAndDoEMFRegistration();
	}

	@Override
	public void register(Injector injector) {
		// added after the switching to Xcore
		if (!EPackage.Registry.INSTANCE.containsKey("http://www.xtext.org/example/helloxcore/HelloXcore")) {
			EPackage.Registry.INSTANCE.put("http://www.xtext.org/example/helloxcore/HelloXcore", org.xtext.example.helloxcore.helloxcore.HelloxcorePackage.eINSTANCE);
		}

		super.register(injector);
	}
}

And now the Junit tests will run again.

Modifying the metamodel with Xcore

We can now customize our metamodel, using the Xcore editor.

For example, we add the interface Element, with the method getName() and we make both Hello and Greeting implement this interface (they both have getName() thus the implementation of the interface is automatic).

interface Element {
	op String getName()
}

class Hello extends Element {
	String name
}

class Greeting extends Element {
	String name
	contains XExpression expression
}

Using the Xcore editor is easy, and you have content assist; as soon as you press save, the Java files will be automatically regenerated:

xcore_modify1

We also add a method getElements() to the Model class returning an Iterable<Element>(containing both the Hello and the Greeting objects). This time, with Xcore, it is really easy to do so (compare that with the procedure of the previous post, requiring the use of EAnnotation in the Ecore file), since Xcore uses Xbase expression syntax for defining the body of the operations (with full content assist, not to mention automatic import statement insertions). See also the generated Java code on the right:

xcore_modify2

And now we can implement the validator method checking duplicates, using the new getElements() method and the fact that now both Hello and Greeting implement Element:

package org.xtext.example.helloxcore.validation

import org.eclipse.xtext.validation.Check
import org.eclipse.xtext.xbase.typesystem.util.Multimaps2
import org.xtext.example.helloxcore.helloxcore.Element
import org.xtext.example.helloxcore.helloxcore.Model

class HelloXcoreValidator extends AbstractHelloXcoreValidator {

	public static val DUPLICATE_NAME = 'HelloXcoreDuplicateName'

	@Check
	def checkDuplicateElements(Model model) {
		val nameMap = <String,Element>Multimaps2.newLinkedHashListMultimap
		
		for (e : model.elements)
			nameMap.put(e.name, e)
		
		for (entry : nameMap.asMap.entrySet) {
			val duplicates = entry.value
			if (duplicates.size > 1) {
				for (d : duplicates)
					error(
						"Duplicate name '" + entry.key + "' (" + d.eClass.name + ")",
						d,
						null, 
						DUPLICATE_NAME);
			}
		}
	}
}

That’s all! I hope you found this tutorial useful :)

 

Be Sociable, Share!

Switching from an inferred Ecore model to an imported one in your Xtext grammar

When you use Xtext for developing your language the Ecore model for the AST is automatically derived/inferred from the grammar. If your DSL is simple, this automatic meta-model inference is usually enough. However, there might be cases where you need more control on the meta-model and in such cases you will want to switch from an inferred Ecore model to a an imported one, which you will manually maintain. This is documented in the Xtext documentation, and in some blog posts. When I needed to switch to an imported Ecore model for Xsemantics, things have not been that easy, so I thought to document the steps to perform some switching in this tutorial, using a simple example. (I should have talked about that in my Xtext book, but at that time I ran out of pages so there was no space left for this subject :)

So first of all, let’s create an Xtext project, org.xtext.example.hellocustomecore, (you can find the sources of this example online at https://github.com/LorenzoBettini/Xtext2-experiments); the grammar of the DSL is not important: this is just an example. We will first start developing the DSL using the automatic Ecore model inference and later we will switch to an imported Ecore.

The grammar of this example is as follows (to make things more interesting, we will also use Xbase):

grammar org.xtext.example.hellocustomecore.HelloCustomEcore with 
	org.eclipse.xtext.xbase.Xbase

generate hellocustomecore "http://www.xtext.org/example/hellocustomecore/HelloCustomEcore"

Model:
	importSection=XImportSection?
	hellos+=Hello*
	greetings+=Greeting*
;

Hello:
    'Hello' name=ID '!'
;

Greeting:
    'Greeting' name=ID expression = XExpression
;

and we run the MWE2 generator.

To have something working, we also write an inferrer

def dispatch void infer(Model element, IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
   		acceptor.accept(element.toClass("generated." + element.eResource.URI.lastSegment.split("\\.").head.toFirstUpper))
   			.initializeLater([
   				for (hello: element.hellos) {
   					members += hello.toMethod
   							("say" + hello.name.toFirstUpper, hello.newTypeRef(String)) [
   						body = '''return "Hello «hello.name»";'''
   					]
   				}
   				for (greeting : element.greetings) {
   					members += greeting.toMethod
   							("say" + greeting.name.toFirstUpper, greeting.newTypeRef(String)) [
   						body = greeting.expression
   					]
   				}
   			])
   	}

With this DSL we can write programs of the shape (nothing interesting, this is just an example)

Hello foo!

Greeting bar {
	sayFoo() + "bar"
}

Now, let’s say we want to check in the validator that there are no elements with the same name; since both “Hello” and “Greeting” have the feature name, we can introduce in the Ecore model a common interface with the method getName(). OK, we could achieve this also by introducing a fake rule in the Xtext grammar, but let’s switch to an imported Ecore model so that we can manually modify that.

Switching to an imported Ecore model

First of all, we add a new source folder to our project, say emf-gen, where all the EMF classes will be generated; we also make sure to include such folder in the build.properties file:

source.. = src/,\
           src-gen/,\
           xtend-gen/,\
           emf-gen/
bin.includes = model/,\
               META-INF/,\
               .,\
               plugin.xml

Remember that, at the moment, the EMF classes are generated into the src-gen folder, together with other Xtext artifacts (e.g., the ANTLR parser):

imported-ecore-project-layout1

Xtext generates the inferred Ecore model file and the GenModel file into the folder model/generated

imported-ecore-project-layout2

This is the new behavior introduced in Xtext 2.4.3 by the fragment ecore.EMFGeneratorFragment that replaces the now deprecated ecore.EcoreGeneratorFragment; if you still have the deprecated fragment in your MWE2 files, then the Ecore and the GenModel are generated in the src-gen folder.

Let’s rename the “generated” folder into “custom” (if in the future for any reason we want to re-enable Xtext Ecore inference, our custom files will not be overwritten):

imported-ecore-project-layout3

NOTE: if you simply move the .ecore and .genmodel file into the directory model, you will not be able to open the .ecore file with the Ecore editor: this is due to the fact that this Ecore file refers to Xbase Ecore models with a relative path; in that case you need to manually adjust such references by opening the .ecore file with the text editor.

From now on, remember, we will manually manage the Ecore file.

Now we change the GenModel file, so that the EMF model classes are generated into emf-gen instead of src-gen:

imported-ecore-genmodelWe need to change the MWE2 file as follows:

  • Enable the org.eclipse.emf.mwe2.ecore.EcoreGenerator fragment that will generate the EMF classes using our custom Ecore file and GenModel file; indeed, you must refer to the custom GenModel file; before that we also run the DirectoryCleaner on the emf-gen folder (this way, each time the EMF classes are generated, the previous classes are wiped out); enable these two parts right after the StandaloneSetup section;
  • Comment or remove the DirectoryCleaner element for the model directory (otherwise the workflow will remove our custom Ecore and GenModel files);
  • In the language section we load our custom Ecore file,
  • and we disable ecore.EMFGeneratorFragment (we don’t need that anymore, since we don’t want the Ecore model inference)

The MWE2 files is now as follows (I highlighted the modifications):

...
Workflow {
    bean = StandaloneSetup {
    	scanClassPath = true
    	platformUri = "${runtimeProject}/.."
    	// The following two lines can be removed, if Xbase is not used.
    	registerGeneratedEPackage = "org.eclipse.xtext.xbase.XbasePackage"
    	registerGenModelFile = "platform:/resource/org.eclipse.xtext.xbase/model/Xbase.genmodel"
    }

	component = DirectoryCleaner {
		directory = "${runtimeProject}/emf-gen"
	}

	component = org.eclipse.emf.mwe2.ecore.EcoreGenerator {
		genModel = "platform:/resource/${projectName}/model/custom/HelloCustomEcore.genmodel"
		srcPath = "platform:/resource/${projectName}/src"
	}

    component = DirectoryCleaner {
    	directory = "${runtimeProject}/src-gen"
    }

    //component = DirectoryCleaner {
    //	directory = "${runtimeProject}/model"
    //}

    component = DirectoryCleaner {
    	directory = "${runtimeProject}.ui/src-gen"
    }

    component = DirectoryCleaner {
    	directory = "${runtimeProject}.tests/src-gen"
    }

    component = Generator {
    	pathRtProject = runtimeProject
    	pathUiProject = "${runtimeProject}.ui"
    	pathTestProject = "${runtimeProject}.tests"
    	projectNameRt = projectName
    	projectNameUi = "${projectName}.ui"
    	encoding = encoding
    	language = auto-inject {
    		loadedResource = "platform:/resource/${projectName}/model/custom/HelloCustomEcore.ecore"

    		uri = grammarURI

    		// Java API to access grammar elements (required by several other fragments)
    		fragment = grammarAccess.GrammarAccessFragment auto-inject {}

    		// generates Java API for the generated EPackages
    		// fragment = ecore.EMFGeneratorFragment auto-inject {}

    		// the old serialization component
    		// fragment = parseTreeConstructor.ParseTreeConstructorFragment auto-inject {}    

    		// serializer 2.0
    		fragment = serializer.SerializerFragment auto-inject {
    			generateStub = false
    		}
...

We add the dependency org.eclipse.xtext.ecore in the MANIFEST.MF:imported-ecore-manifestIn the Xtext grammar we replace the generate statement with an import statement:

grammar org.xtext.example.hellocustomecore.HelloCustomEcore with 
	org.eclipse.xtext.xbase.Xbase

//generate hellocustomecore "http://www.xtext.org/example/hellocustomecore/HelloCustomEcore"
import "http://www.xtext.org/example/hellocustomecore/HelloCustomEcore"

Now we’re ready to run the MWE2 workflow, and you should get no error (if you followed all the above instructions); you can see that now the EMF model classes are generated into the emf-gen folder (the corresponding packages in the src-gen folders are now empty and you can remove them):

imported-ecore-project-layout4

We must now modify the plugin.xml (note that there’s no plugin.xml_gen anymore), so that the org.eclipse.emf.ecore.generated_package extension point contains the reference to the new GenModel file:

<plugin>

  <extension point="org.eclipse.emf.ecore.generated_package">
    <package 
       uri = "http://www.xtext.org/example/hellocustomecore/HelloCustomEcore" 
       class = "org.xtext.example.hellocustomecore.hellocustomecore.HellocustomecorePackage"
       genModel = "model/custom/HelloCustomEcore.genmodel" /> 

  </extension>

</plugin>

If you try the editor for the DSL it will still work; however, the Junit tests will fail with errors of this shape:

org.eclipse.xtext.parser.ParseException: 
java.lang.IllegalStateException: 
Unresolved proxy http://www.xtext.org/example/hellocustomecore/HelloCustomEcore#//Hello. 
Make sure the EPackage has been registered

That’s because the generated StandaloneSetup does not register the EPackage anymore, see the diff:

imported-ecore-diff

All we need to do is to modify the StandaloneSetup in the src folder (NOT the generated one, since it will be overwritten by subsequent MWE2 workflow runs) and override the register method so that it performs the registration of the EPackage:

public class HelloCustomEcoreStandaloneSetup extends HelloCustomEcoreStandaloneSetupGenerated{

	public static void doSetup() {
		new HelloCustomEcoreStandaloneSetup().createInjectorAndDoEMFRegistration();
	}

	@Override
	public void register(Injector injector) {
		if (!EPackage.Registry.INSTANCE.containsKey("http://www.xtext.org/example/hellocustomecore/HelloCustomEcore")) {
			EPackage.Registry.INSTANCE.put("http://www.xtext.org/example/hellocustomecore/HelloCustomEcore", org.xtext.example.hellocustomecore.hellocustomecore.HellocustomecorePackage.eINSTANCE);
		}
		super.register(injector);
	}
}

And now the Junit tests will run again.

Modifying the Ecore model

We can now customize our Ecore model, using the Ecore editor and the Properties view.

For example, we add the interface Element, with the method getName() and we make both Hello and Greeting implement this interface (they both have getName() thus the implementation of the interface is automatic).

imported-ecore-custom-ecore1 imported-ecore-custom-ecore2 imported-ecore-custom-ecore3

We also add a method getElements() to the Model class returning an Iterable<Element> (containing both the Hello and the Greeting objects)

imported-ecore-custom-ecore4

and we implement that method using an EAnnotation, using the source “http://www.eclipse.org/emf/2002/GenModel” and providing a body

imported-ecore-custom-ecore5 imported-ecore-custom-ecore6

With the following implementation

return 
	com.google.common.collect.Iterables.<Element>concat
		(getHellos(), getGreetings());

Let’s run the MWE2 workflow so that it will regenerate the EMF classes.

And now we can implement the validator method checking duplicates, using the new getElements() method and the fact that now both Hello and Greeting implement Element:

import org.eclipse.xtext.validation.Check
import org.eclipse.xtext.xbase.typesystem.util.Multimaps2
import org.xtext.example.hellocustomecore.hellocustomecore.Element
import org.xtext.example.hellocustomecore.hellocustomecore.Model

class HelloCustomEcoreValidator extends AbstractHelloCustomEcoreValidator {

	public static val DUPLICATE_NAME = 'HelloCustomEcoreDuplicateName'

	@Check
	def checkDuplicateElements(Model model) {
		val nameMap = <String,Element>Multimaps2.newLinkedHashListMultimap

		for (e : model.elements)
			nameMap.put(e.name, e)

		for (entry : nameMap.asMap.entrySet) {
			val duplicates = entry.value
			if (duplicates.size > 1) {
				for (d : duplicates)
					error(
						"Duplicate name '" + entry.key + "' (" + d.eClass.name + ")",
						d,
						null, 
						DUPLICATE_NAME);
			}
		}
	}
}

That’s all! I hope you found this tutorial useful :)

Be Sociable, Share!

Using the Xtend compiler in Buckminster builds

Up to now, I was always putting the Xtend generated Java files in my git repositories (for my Xtext projects), since I still hadn’t succeeded in invoking the Xtend standalone compiler in a Buckminster build. Dennis Hübner published a post with some hints on how to achieve that, but that never worked for me (and apparently it did not work for other users).

After some experiments, it seems I finally managed to trigger Xtend compilation in Buckminster builds, and in this post I’ll show the steps to achieve that (I’m using an example you can find on Github).

The main problems I had to solve were:

  • how to pass the classpath to the Xtend compiler
  • how to deal with chicken-and-egg problems (dependencies among Java and Xtend classes).

IMPORTANT: the build process described here uses a new flag for the Buckminster’s build command, which has been recently added; thus, you must make sure you have an updated version of Buckminster headless (from 4.3 repository).

The steps to perform can be applied to your projects as well; they are simple and easy to reproduce. In this blog post I’ll try to explain them in details.

This blog post assumes that you are already familiar with setting up a Buckminster build.

The example

The example I’m using is an Xtext DSL (just the Greeting example using Xbase), with many .xtend files and with the standard structure:

  • org.xtext.example.hellobuck, the runtime plugin,
  • org.xtext.example.hellobuck.ui, the ui plugin, which uses Xtend classes defined in the runtime plugin,
  • org.xtext.example.hellobuck.tests, the tests plugin, which uses Xtend classes defined in the runtime and in the ui plugin,
  • org.xtext.example.hellobuck.sdk, the SDK feature for the DSL.

Furthermore, we have two additional projects created by the Xtext Buckminster Wizard:

  • org.xtext.example.hellobuck.buckminster, the releng project,
  • org.xtext.example.hellobuck.site, the feature project for creating the p2 repository,

I blogged about the Xtext Buckminster Wizard in the past, and indeed this example is a fork of the example presented in that blog post.

Creating a launch configuration for the Xtend compiler

The first step consists in creating a Java launch configuration in the runtime plugin project that invokes the Xtend standalone compiler. This was shown in Dennis’ original post, but you need to change a few things. Here’s the XtendCompiler.launch file to put in the org.xtext.example.hellobuck runtime plugin project (of course you can call the launch file whateven you want):

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<launchConfiguration type="org.eclipse.jdt.launching.localJavaApplication">
<listAttribute key="org.eclipse.debug.core.MAPPED_RESOURCE_PATHS">
<listEntry value="/org.xtext.example.hellobuck"/>
</listAttribute>
<listAttribute key="org.eclipse.debug.core.MAPPED_RESOURCE_TYPES">
<listEntry value="4"/>
</listAttribute>
<stringAttribute key="org.eclipse.jdt.launching.MAIN_TYPE" value="org.eclipse.xtend.core.compiler.batch.Main"/>
<stringAttribute key="org.eclipse.jdt.launching.PROGRAM_ARGUMENTS" value="-classpath ${project_classpath:org.xtext.example.hellobuck} -d xtend-gen src"/>
<stringAttribute key="org.eclipse.jdt.launching.PROJECT_ATTR" value="org.xtext.example.hellobuck"/>
</launchConfiguration>

This launch configuration can be reused in other projects, provided the highlighted lines are changed accordingly, since they refer to the containing project.

An important part of this launch configuration is the PROGRAM_ARGUMENTS that are passed to the Xtend compiler, in particular the -classpath argument. This was the main problem I experienced in the past (and that I saw in all the other posts in the forum): the Xtend compiler needs to find the Java classes your Xtend files depend upon and thus you need to pass a valid -classpath argument. But we can simply reuse the classpath of the containing project :)

-classpath ${project_classpath:org.xtext.example.hellobuck}

Add dependency for Xtend standalone compiler

This launch configuration calls the Java application org.eclipse.xtend.core.compiler.batch.Main thus you must add a dependency on the corresponding bundle in your MANIFEST.MF. The bundle you need to depend on is org.eclipse.xtend.standalone (the dependency can be optional):

xtend_standalone_dependency

 

Test the launch in your workbench

You can test this launch configuration from Eclipse, with Run As => Java Application. In the Console view you should see something like:

0    [main] INFO  e.compiler.batch.XtendBatchCompiler  - Compiling 4 source files to xtend-gen
0 [main] INFO org.eclipse.xtend.core.compiler.batch.XtendBatchCompiler  - Compiling 4 source files to xtend-gen

This will give you confidence that the launch configuration works correctly and that all dependencies for invoking the Xtend compiler are in place.

Add an XtendCompiler.launch in the other projects

You must now add an XtendCompiler.launch in all the other projects containing Xtend files. In our example we must add it to the ui and the tests projects.

You can copy the one you have already created but MAKE SURE you update the relevant 3 parts according to the containing projects! See the highlighted lines above.

NOTE: you do NOT need to add a dependency on org.eclipse.xtend.standalone in the MANIFEST.MF of the ui and tests projects: they depend on the runtime plugin project which already has that dependency.

You may want to run the XtendCompiler.launch also in these projects from the Eclipse workbench, again to get confidence that you configured the launch configurations correctly.

IMPORTANT: when the Xtend compiler compiles the files in the ui and tests project, you will see some ERROR lines, e.g.,

0    [main] ERROR pes.access.impl.DeclaredTypeFactory  - Incomplete methods for org.eclipse.xtext.xbase.ui.labeling.XbaseImages2: java.lang.NoClassDefFoundError: org/eclipse/jdt/ui/JavaElementImageDescriptor
1    [main] ERROR pes.access.impl.DeclaredTypeFactory  - Incomplete constructors for org.eclipse.xtext.xbase.ui.labeling.XbaseImages2: java.lang.NoClassDefFoundError: org/eclipse/jdt/ui/JavaElementImageDescriptor
1664 [main] ERROR pes.access.impl.AbstractClassMirror  - resource is empty: java:/Objects/org.eclipse.core.runtime.IPluginDescriptor
1718 [main] ERROR pes.access.impl.AbstractClassMirror  - resource is empty: java:/Objects/org.eclipse.core.runtime.dynamichelpers.IExtensionTracker
3513 [main] INFO  e.compiler.batch.XtendBatchCompiler  - Compiling 6 source files to xtend-gen
3513 [main] INFO org.eclipse.xtend.core.compiler.batch.XtendBatchCompiler  - Compiling 6 source files to xtend-gen

From what I understand, these errors do not prevent the Xtend compiler to successfully generate Java files (see the final INFO line) and the procedure terminates successfully. Thus, you can ignore these errors. If the Xtend compiler really cannot produce Java files it will terminate with a final error.

Configure the headless build

Now it’s time to configure the Buckminster headless build so that it runs the Xtend compiler. We created .launch files because one of the cool things of Buckminster is that it can seamlessly run the launch files.

The tricky part here is that since we perform a clean build, there is a chicken-and-egg scenario

  • no Java files have been compiled,
  • most Java files import Java files created by Xtend
  • the Xtend files import Java classes

To solve these problems we perform an initial clean build; this will run the Java compiler and such compilation will terminate with errors. We expect that, due to the chicken-and-egg situation. However, this will create enough .class files to run the Xtend compiler! It is important to run the build command with the (new) flag –continueonerror, otherwise the whole build will fail.

After running XtendCompiler.launch in the org.xtext.example.hellobuck runtime project, we run another build –continueonerror so that the Java files generated by the Xtend compiler will be compiled by Java. We then proceed similarly for the ui and the tests project:

clean

# this first Java build will fail with compilation errors, since Xtend
# classes have not been compiled yet.  However, it will compile some
# Java classes so that we will be able to compile Xtend classes
build --continueonerror

launch --stderr --stdout -l "org.xtext.example.hellobuck/XtendCompiler.launch"
build --continueonerror
# you need to run a Java build to compile the Java files generated by
# the Xtend compiler.  They will be needed for compiling Xtend classes
# in the UI plugin.

launch --stderr --stdout -l "org.xtext.example.hellobuck.ui/XtendCompiler.launch"
build --continueonerror
# you need to run a Java build to compile the Java files generated by
# the Xtend compiler.  They will be needed for compiling Xtend classes
# in the TESTS plugin.

launch --stderr --stdout -l "org.xtext.example.hellobuck.tests/XtendCompiler.launch"

Then, your build can proceed as usual (at this point I prefer to perform a clean build): run the tests (both plain Junit and Plug-in Junit tests) and create the p2 repository:

build --clean

junit -l "org.xtext.example.hellobuck.tests/org.xtext.example.hellobuck.tests.launch"  --flatXML --output "${buckminster.output.root}/test.results/org.xtext.example.hellobuck.tests.launch.xml"
junit -l "org.xtext.example.hellobuck.tests/org.xtext.example.hellobuck.ui.tests.launch"  --flatXML --output "${buckminster.output.root}/test.results/org.xtext.example.hellobuck.ui.tests.launch.xml"

perform "org.xtext.example.hellobuck.site#site.p2"

The complete commands file can be seen here.

Executing the headless build

You can now run your headless build on your machine or on Jenkins.

My favorite way of doing that is by using an ANT script. The whole ANT script can be found in the example.

This script also automatically installs Buckminster headless if not present.

Before executing the commands file, it also removes the contents of the xtend-gen folder in all the projects; this way you are sure that no stale generated Java files are there.

Of course, you should now remove the xtend-gen folder from your git repository (and put it in the .gitignore file).

In Jenkins you can configure an Invoke Ant build step as shown in the screenshot (“Start Xvfb before the build, and shut it down after” is required to execute Plug-in Junit tests; we also pass an option to install Buckminster headless in the job’s workspace).

xtext-xtend-buckminster-jenkins

Try the example

You can just clone the git repository, and then

cd org.xtext.example.hellobuck.buckminster
ant -f build.ant

As noted above, this will also install Buckminster headless if not found in the location specified by the property buckminster.home. This script will take some time, especially the first time, since it will materialize the target platform.

Hope you find this blog post useful! :)

Be Sociable, Share!

The book on Xtext is out

My book on Xtext, “Implementing Domain-Specific Languages with Xtext and Xtend” is now available on Packt website! Get it while it’s hot! :)

You can find the outline and an example chapter at

http://www.packtpub.com/implementing-domain-specific-languages-with-xtext-and-xtend/book

Many thanks to the reviewers of the book: Jan Koehnlein, Henrik Lindberg, Pedro J. Molina, and Sebastian Zarnekow!

The sources of the examples presented in the book are available at https://github.com/LorenzoBettini/packtpub-xtext-book-examples

0304OS_mockupcover_normalI would also like to thank all the people from Packt I dealt with.

 

Be Sociable, Share!

The XImportSection in Xbase 2.4

I know that Xtext 2.4 has not been released yet, but I could not resist blogging about a very cool new feature in Xbase: improved automatic import functionalities!

Actually, import functionalities were already great when using Xbase also in previous versions of Xtext, but now they provide a much better experience for the user of your DSL! Indeed, all the import functionalities you are used to with JDT (like automatic import insertion, and organize imports) are available also for your Xbase language; these features were already available in Xtend, and they have been ported to Xbase itself.

At the time of writing, you need to get the very latest updates of Xtext 2.4, using the update site http://download.eclipse.org/modeling/tmf/xtext/updates/composite/latest/ .

Before you used to do something like

grammar org.xtext.example.helloinferrer.HelloInferrer with 
	org.eclipse.xtext.xbase.Xbase

generate helloInferrer "http://www.xtext.org/example/helloinferrer/HelloInferrer"

Model:
	imports += Import*
	greetings+=Greeting*;

Import:
  'import' importedNamespace = QualifiedNameWithWildcard
;

...

Now, you can use in your grammar the new Xbase rule: XImportSection:

grammar org.xtext.example.helloinferrer.HelloInferrer with 
	org.eclipse.xtext.xbase.Xbase

generate helloInferrer "http://www.xtext.org/example/helloinferrer/HelloInferrer"

Model:
	importSection=XImportSection?
	greetings+=Greeting*;

...

In this post I’m reusing some experiments you can find here (https://github.com/LorenzoBettini/Xtext2-experiments, I had blogged about these experiments in previous posts).

If you now rerun the MWE2 generator, and make sure you merge the plugin.xml_gen with plugin.xml in the .ui project, your editor will provide some interesting features for free (if you use my examples, you can find a project wizard “New Project” => “Xtext” => “HelloInferrer Project”):

Imports with wildcards are deprecated:

use_of_wildcards_deprecated

You now have the context menu “Organize Imports” (Shift + Control + O); try that one in the presence of such deprecation warning and imports are organized for you:

after_organize_imports

Similarly, unused imports are reported as warnings:

imports_never_used

Again, use “Organize Imports” to fix that!

The new feature I like most is the automatic insertion of imports! (just like in JDT and Xtend): try to get content assist for a Java type, for instance,

code_completion

Accept a proposal and the import will be automatically inserted (instead of the fully qualified name):

code_completion2

Xtext rocks! :)

Be Sociable, Share!

Building Xtext projects with Buckminster

One of the new cool features which came with Xtext 2.3.0 is a wizard which generates all the artifacts to build your Xtext DSL project with Buckminster; this will allow you to easily build your Xtext project p2 site and also to build your project and its tests headlessly (e.g., from the command line, and, more importantly from within a continuous integration system like Jenkins).

This new feature was not advertised much, nor documented, so I decided to write a tutorial about that, also with the permission of the developer of this feature Dennis Hübner.

In this tutorial I will show how to

  • create the p2 repository from the IDE
  • build your project (and run the tests) headlessly from the command line (through ant)
  • build your project (and run the tests) headlessly in a continuous integration system like Jenkins

In the end, I will also try to describe/explain all the Buckminster files that the wizard created for you, so that one can customize them.

The sources used in this tutorial can be found at

https://github.com/LorenzoBettini/Xtext-Buckminster-Example

Create an Xtext project

So, first of all, let’s create a simple Xtext hello project; for this example I will use the following settings:

  • project name: org.xtext.example.hellobuck
  • name: org.xtext.example.hellobuck.HelloBuck
  • extension: greetings
  • check “Create SDK feature project”

and then, of course, we generate Xtext artifacts.

The DSL is basically the standard Greetings DSL; in this case, I’m using the JvmModelInferrer to generate Java classes from “Hello” specifications. Indeed, the DSL itself is not important in this tutorial. I’ve also added two Junit tests in the corresponding org.xtext.example.hellobuck.tests project:

Note the org.xtext.example.hellobuck.tests.launch Launch configuration that Xtext created for you (which basically runs all the Junit tests in this project. We will see how this will be used later.

Use the Xtext Build with Buckminster Wizard

Now we can use the Xtext wizard “Build with Buckminster”: New => Other => Xtext => Continuous Integration => Build with Buckminster:

You will have to specify the sdk feature of your Xtext project (that has been created by the Xtext new project wizard); the releng and site projects will have predefined names. For the Buckminster installation directory you have 3 choices:

  • if you have already an headless installation of Buckminster in your system you can specify the path;
  • you can install it using the link in the dialog
  • you can leave that path empty and specify it later in the build.ant generated file (or pass it on the command line when invoking ant)

In any case, specifying the headless Buckminster installation path is only useful if you intend to build your projects headlessly with ant (it is not requested for building in the IDE, nor if you plan to build it with Jenkins by setting job using the Buckminster Jenkins plugin, as we will see later).

Before pressing Finish, we also want to specify the tests to launch during the headless build, so we press the button Add and we select the launch configuration in the .tests project which the Xtext wizard created for us.

If you want to build an existing Xtext project and you do not have that launch configuration, all you need to do is to simply create a new plain Junit launch configuration from the IDE, for instance, to run your existing Junit test suite, and save it in the .tests project yourself. You can also configure the headless build to run tests later on.

When the wizard finishes, you will end up with two additional projects

  • org.xtext.example.hellobuck.buckminster with all the Buckminster (CQUERY and RMAP, build.ant, etc.) files to build your project headlessly;
  • org.xtext.example.hellobuck.site which is a feature project to create the p2 repository for your Xtext project.

Build the p2 repository from the IDE

Although the .buckminster project created by the wizard is thought to be used in an headless environment, the .site project is useful also to build the p2 repository from the IDE itself! Indeed, if you created an old style update site, you could build the update site from the IDE itself. But if you switched to the new category.xml format, then building the corresponding p2 repository from the IDE is not straightforward.

If you want to run Buckminster from Eclipse, you first need to install it in the IDE of course, by using this repository: http://download.eclipse.org/tools/buckminster/updates-4.2

Before creating the p2 repository, you may want to tweak some Buckminster properties, for instance, you may want to create a buckminster.properties file in the .buckminster project (or in the .site project) like the following

# Where all the output should go
buckminster.output.root=${user.home}/tmp/hellobuck/build
# Where the temp files should go
buckminster.temp.root=${user.home}/tmp/hellobuck
# How .qualifier in versions should be replaced
qualifier.replacement.*=generator:lastRevision

target.os=*
target.ws=*
target.arch=*

so that Buckminster will generate all its artifacts in the tmp/hellobuck directory of your home folder.

We now right click on the .site project, and select Buckminster => Invoke Action… , we select site.p2 action, and optionally refer to the buckminster.properties file

When the action finishes, you will file the p2.site in the directory $HOME/tmp/hellobuck/build/org.xtext.example.hellobuck.site_1.0.0-eclipse.feature/site.p2/

Build with ant

The wizard created for us, in the .buckminster project a build.ant file that we can use to build the project, run the tests, and create the p2 repository with ant

Note that this file has the location of your Buckminster headless installation path hardcoded (recall the wizard we used before for creating Buckminster projects); if that is not correct, you can still pass this information to ant when invoking it

ant -Dbuckminster.home=<PATH> -f build.ant

This ant script will build your project in the directory buildroot which will be created in the directory where all your projects are located (e.g., in the workspace or in another path); in particular it will

  1. materialize the target platform for your project (the first time you run it), and this requires an Internet connection and might take some time
  2. build your projects
  3. run the Junit tests
  4. build the p2 site repository

This should be the output (where WORKSPACE is actually the location of your projects):

Buildfile: WORKSPACE/org.xtext.example.hellobuck.buckminster/build.ant
cleanup:
buckminster:
     [echo] IMPORTANT: Populating an empty target platform may took over 10 minutes.
     [exec] INFO:  setpref 'targetPlatformPath=WORKSPACE/buildroot/target.platform'
     [exec] WARN:  Target platform directory 'WORKSPACE/buildroot/target.platform' does not exist and will be created
     [exec] INFO:  resolve 'WORKSPACE/org.xtext.example.hellobuck.buckminster/projects-platform.cquery'
     [exec] INFO:  Resetting target platform Directory WORKSPACE/buildroot/target.platform
     [exec] INFO:  Import complete.
     [exec] INFO:  resolve 'WORKSPACE/org.xtext.example.hellobuck.buckminster/project.cquery'
     [exec] INFO:  Import complete.
     [exec] INFO:  build
     [exec] INFO:  junit '-l' 'org.xtext.example.hellobuck.tests/org.xtext.example.hellobuck.tests.launch' '--flatXML' '--output' 'WORKSPACE/buildroot/buckminster.output/test.results/org.xtext.example.hellobuck.tests.launch.xml'
     [exec] INFO:  Starting test session org.xtext.example.hellobuck.tests
     [exec] INFO:  Running test org.xtext.example.hellobuck.tests.HelloBuckCompilerTest.testGeneratedJava...
     [exec] INFO:    ...OK [0.877s]
     [exec] INFO:  Running test org.xtext.example.hellobuck.tests.HelloBuckCompilerTest.compareGeneratedJava...
     [exec] INFO:    ...OK [0.189s]
     [exec] INFO:  Running test org.xtext.example.hellobuck.tests.HelloBuckParserTest.testParsingAndValidation...
     [exec] INFO:    ...OK [0.096s]
     [exec] INFO:  Tests finished.
     [exec] INFO:  Elapsed time: 1.164 seconds.
     [exec] INFO:  Total number of tests executed: 3
     [exec] INFO:  Successful tests: 3
     [exec] INFO:  Failed tests: 0
     [exec] INFO:  Errors: 0
     [exec] INFO:  Ignored tests: 0
     [exec] INFO:  Overall status: OK
     [exec] INFO:  perform 'org.xtext.example.hellobuck.site#site.p2'
     [exec] INFO:  [start org.xtext.example.hellobuck:osgi.bundle$1.0.0.qualifier#eclipse.build]
     [exec] INFO:  [end org.xtext.example.hellobuck:osgi.bundle$1.0.0.qualifier#eclipse.build]
     [exec] INFO:  [start org.xtext.example.hellobuck:osgi.bundle$1.0.0.qualifier#manifest]
     [exec] INFO:  [end org.xtext.example.hellobuck:osgi.bundle$1.0.0.qualifier#manifest]
     [exec] INFO:  [start org.xtext.example.hellobuck:osgi.bundle$1.0.0.qualifier#bundle.jar]
     [exec] INFO:  [end org.xtext.example.hellobuck:osgi.bundle$1.0.0.qualifier#bundle.jar]
     [exec] INFO:  [start org.xtext.example.hellobuck.ui:osgi.bundle$1.0.0.qualifier#eclipse.build]
     [exec] INFO:  [end org.xtext.example.hellobuck.ui:osgi.bundle$1.0.0.qualifier#eclipse.build]
     [exec] INFO:  [start org.xtext.example.hellobuck.ui:osgi.bundle$1.0.0.qualifier#manifest]
     [exec] INFO:  [end org.xtext.example.hellobuck.ui:osgi.bundle$1.0.0.qualifier#manifest]
     [exec] INFO:  [start org.xtext.example.hellobuck.ui:osgi.bundle$1.0.0.qualifier#bundle.jar]
     [exec] INFO:  [end org.xtext.example.hellobuck.ui:osgi.bundle$1.0.0.qualifier#bundle.jar]
     [exec] INFO:  [start org.xtext.example.hellobuck.sdk:eclipse.feature$1.0.0.qualifier#manifest]
     [exec] INFO:  [end org.xtext.example.hellobuck.sdk:eclipse.feature$1.0.0.qualifier#manifest]
     [exec] INFO:  [start org.xtext.example.hellobuck.sdk:eclipse.feature$1.0.0.qualifier#feature.jar]
     [exec] INFO:  [end org.xtext.example.hellobuck.sdk:eclipse.feature$1.0.0.qualifier#feature.jar]
     [exec] INFO:  [start org.xtext.example.hellobuck:osgi.bundle$1.0.0.qualifier#source.manifest]
     [exec] INFO:  [end org.xtext.example.hellobuck:osgi.bundle$1.0.0.qualifier#source.manifest]
     [exec] INFO:  [start org.xtext.example.hellobuck:osgi.bundle$1.0.0.qualifier#source.bundle.jar]
     [exec] INFO:  [end org.xtext.example.hellobuck:osgi.bundle$1.0.0.qualifier#source.bundle.jar]
     [exec] INFO:  [start org.xtext.example.hellobuck.ui:osgi.bundle$1.0.0.qualifier#source.manifest]
     [exec] INFO:  [end org.xtext.example.hellobuck.ui:osgi.bundle$1.0.0.qualifier#source.manifest]
     [exec] INFO:  [start org.xtext.example.hellobuck.ui:osgi.bundle$1.0.0.qualifier#source.bundle.jar]
     [exec] INFO:  [end org.xtext.example.hellobuck.ui:osgi.bundle$1.0.0.qualifier#source.bundle.jar]
     [exec] INFO:  [start org.xtext.example.hellobuck.sdk:eclipse.feature$1.0.0.qualifier#source.manifest]
     [exec] INFO:  [end org.xtext.example.hellobuck.sdk:eclipse.feature$1.0.0.qualifier#source.manifest]
     [exec] INFO:  [start org.xtext.example.hellobuck.sdk:eclipse.feature$1.0.0.qualifier#source.feature.jar]
     [exec] INFO:  [end org.xtext.example.hellobuck.sdk:eclipse.feature$1.0.0.qualifier#source.feature.jar]
     [exec] INFO:  [start org.xtext.example.hellobuck.site:eclipse.feature$1.0.0.qualifier#copy.subfeatures]
     [exec] INFO:  [end org.xtext.example.hellobuck.site:eclipse.feature$1.0.0.qualifier#copy.subfeatures]
     [exec] INFO:  [start org.xtext.example.hellobuck.site:eclipse.feature$1.0.0.qualifier#copy.plugins]
     [exec] INFO:  [end org.xtext.example.hellobuck.site:eclipse.feature$1.0.0.qualifier#copy.plugins]
     [exec] INFO:  [start org.xtext.example.hellobuck.site:eclipse.feature$1.0.0.qualifier#manifest]
     [exec] INFO:  [end org.xtext.example.hellobuck.site:eclipse.feature$1.0.0.qualifier#manifest]
     [exec] INFO:  [start org.xtext.example.hellobuck.site:eclipse.feature$1.0.0.qualifier#site.p2]
     [exec] INFO:  [end org.xtext.example.hellobuck.site:eclipse.feature$1.0.0.qualifier#site.p2]
     [echo]  
     [echo] Updatesite output in: WORKSPACE/buildroot/buckminster.output/org.xtext.example.hellobuck.site_*-eclipse.feature/site.p2/
BUILD SUCCESSFUL
Total time: 3 minutes 39 seconds

It will also tell you where you can find the generated p2 site.

IMPORTANT: there’s a bug in the currently generated build.ant file, which prevent you from building with ant in Windows; until the fixed version of the wizard is released, you need to make sure that the build.ant has the right contents, as in this example source.

Build with Jenkins

To build your Xtext project in Jenkins you can either create a Job which uses ant and the build.ant script, or create a Job which uses the Buckminster Jenkins plugin (I’ve also blogged about that). I will detail both ways. Both jobs have in common the access to the git repository

A Job using ant

This requires that you configured an ant installation in your Jenkins, and that you already have a working version of Buckminster headless in Jenkins; you need to specify them in the ant build step configuration (the screeshots, of course, refer to the paths and values for my Jenkins installation):

then, we configure post build actions to archive both the artifacts (the p2 repository) and the Junit results:

A job using Buckminster plugin

To configure a Buckminster build step, we prepare a text file in the .buckminster projects with the Buckminster commands to execute (we take inspiration from the commands.txt file that the wizard created), we call it jenkins-commands.txt:

resolve 'org.xtext.example.hellobuck.buckminster/projects-platform.cquery'
resolve 'org.xtext.example.hellobuck.buckminster/project.cquery'
clean
build
perform -D target.os=* -D target.ws=* -D target.arch=* org.xtext.example.hellobuck.site#buckminster.clean
perform -D target.os=* -D target.ws=* -D target.arch=* org.xtext.example.hellobuck.site#site.p2
junit -l 'org.xtext.example.hellobuck.tests/org.xtext.example.hellobuck.tests.launch' --stderr -o 'tests_output/org.xtext.example.hellobuck.tests.launch.xml'

Then we configure a build step (this relies on a Buckminster installation that we have already configured in Jenkins), note that we refer to the jenkins-commands.txt we created above (an alternative would be to copy the commands directly in the text area “Commands”):

To configure the post build actions, since we used different output paths, we have to specify them accordingly (in particular, we have not used the output directory buildroot, thus the default will be used):

Details and Customization

It may be interesting to learn more about the files that the Xtext Buckminster wizard generated for you (at least, I personally found instructive to look at them, and learned something about Buckminster :) most of which I described in another post of this blog). The details might also help you if you need to customize the generated files.

The .buckminster project contains a Component Specification (CSPEC), buckminster.cspec, which looks like

<?xml version="1.0" encoding="UTF-8"?>
<cs:cspec xmlns:cs="http://www.eclipse.org/buckminster/CSpec-1.0" name="org.xtext.example.hellobuck.buckminster" componentType="buckminster" version="1.0.0">
    <cs:dependencies>
        <cs:dependency name="org.eclipse.platform" componentType="eclipse.feature"/>
        <cs:dependency name="org.eclipse.xtext.sdk" componentType="eclipse.feature"/>
    </cs:dependencies>
</cs:cspec>

thus, due to its dependencies, it represents the target platform for your Xtext project. If your Xtext project required some further specific dependencies to build and test, this is the place to express them!

For instance, in an Xtext project of mine, Xsemantics, I run also some SwtBot tests, which also rely on pde, thus, the dependencies in the corresponding .buckminster project’s buckminster.cspec look like

    <cs:dependencies>
        <cs:dependency name="org.eclipse.platform" componentType="eclipse.feature"/>
        <cs:dependency name="org.eclipse.xtext.sdk" componentType="eclipse.feature"/>
        <cs:dependency name="org.eclipse.swtbot.eclipse" componentType="eclipse.feature"/>
        <cs:dependency name="org.eclipse.swtbot.eclipse.test.junit4" componentType="eclipse.feature"/>
        <cs:dependency name="org.eclipse.swtbot.ide" componentType="eclipse.feature"/>
        <cs:dependency name="org.eclipse.pde" componentType="eclipse.feature"/>
    </cs:dependencies>

To materialize components and dependencies with Buckminster, you need a Resource Map (RMAP), and the wizard created two maps: one containing p2 repositories location (for materializing the features of your target platform), called projects-platform.rmap and one for binding your projects to the workspace (when building headlessly), called project.rmap.

projects-platform.rmap looks like

<?xml version="1.0" encoding="UTF-8"?>
<rm:rmap xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0" xmlns:rm="http://www.eclipse.org/buckminster/RMap-1.0">
  <rm:property key="eclipse.download" value="http://download.eclipse.org"/>
  <rm:property key="xtext.p2.repository" value="${eclipse.download}/modeling/tmf/xtext/updates/composite/releases/"/>
  <rm:property key="eclipse.target.platform" value="${eclipse.download}/releases/juno"/>
  <rm:redirect pattern="org.xtext.example.hellobuck.buckminster" href="project.rmap"/>
  <rm:locator pattern="^org\.eclipse\.(?:xtext|xpand|xtend|xtend2|emf\.mwe|emf\.mwe2)(?:\..+)?$" searchPathRef="xtext"/>
  <rm:locator pattern="^com\.google.*" searchPathRef="xtext"/>
  <rm:locator pattern="^org\.antlr.*" searchPathRef="xtext"/>
  <rm:locator pattern="javax.inject" searchPathRef="xtext"/>
  <rm:locator searchPathRef="eclipse"/>
  <rm:searchPath name="xtext">
    <rm:provider componentTypes="osgi.bundle,eclipse.feature" readerType="p2" source="false">
      <rm:property key="buckminster.source" value="false"/>
      <rm:uri format="{0}">
        <bc:propertyRef key="xtext.p2.repository"/>
      </rm:uri>
    </rm:provider>
  </rm:searchPath>
  <rm:searchPath name="eclipse">
    <rm:provider componentTypes="eclipse.feature,osgi.bundle" readerType="p2" source="false" mutable="false">
      <rm:property key="buckminster.source" value="false"/>
      <rm:property key="buckminster.mutable" value="false"/>
      <rm:uri format="{0}">
        <bc:propertyRef key="eclipse.target.platform"/>
      </rm:uri>
    </rm:provider>
  </rm:searchPath>
</rm:rmap>

This basically tells Buckminster to take all Xtext stuff (including additional components like Google Guice and Antlr) from the Xtext main repository, and to use the main Eclipse Juno release repository for everything else. However, it also redirects to project.rmap for resolving the main Buckminster component of our project org.xtext.example.hellobuck.buckminster.

Again, if your target platform needs additional features from different sites, this is the place where to express them; reusing the example above which also needed SwtBot the rmap would also have these additional stuff:

<?xml version="1.0" encoding="UTF-8"?>
<rm:rmap xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0" xmlns:rm="http://www.eclipse.org/buckminster/RMap-1.0">
  ...
  <rm:property key="swtbot.repository" value="${eclipse.download}/technology/swtbot/helios/dev-build/update-site"/>
  <rm:property key="orbit.repository" value="${eclipse.download}/tools/orbit/downloads/drops/R20120526062928/repository/"/>
  ...
  <rm:locator pattern="^org\.hamcrest*" searchPathRef="orbit"/>
  <rm:locator pattern="^org\.eclipse\.(?:swtbot)(?:\..+)?$" searchPathRef="swtbot"/>
  ...
  <rm:searchPath name="swtbot">
    <rm:provider componentTypes="osgi.bundle,eclipse.feature" readerType="p2" source="false">
      <rm:property key="buckminster.source" value="false"/>
      <rm:uri format="{0}">
        <bc:propertyRef key="swtbot.repository"/>
      </rm:uri>
    </rm:provider>
  </rm:searchPath>
  <rm:searchPath name="orbit">
    <rm:provider componentTypes="osgi.bundle,eclipse.feature" readerType="p2" source="false">
      <rm:property key="buckminster.source" value="false"/>
      <rm:uri format="{0}">
        <bc:propertyRef key="orbit.repository"/>
      </rm:uri>
    </rm:provider>
  </rm:searchPath>
  ...
</rm:rmap>

The project.rmap tells Buckminster where to find the projects of our Xtext project (so that it can bind it to the workspace when building headlessly):

<?xml version="1.0" encoding="UTF-8"?>
<rm:rmap xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0" xmlns:rm="http://www.eclipse.org/buckminster/RMap-1.0">
  <rm:property key="projects.location" value="${workspace.root}"/>
  <rm:locator pattern="^org.xtext.example.hellobuck(?:\..+)?$" searchPathRef="project"/>
  <rm:searchPath name="project">
    <rm:provider componentTypes="eclipse.feature,osgi.bundle,buckminster" readerType="local" mutable="false">
      <rm:property key="buckminster.mutable" value="false"/>
      <rm:uri format="{0}/{1}">
        <bc:propertyRef key="projects.location"/>
        <bc:propertyRef key="buckminster.component"/>
      </rm:uri>
    </rm:provider>
  </rm:searchPath>
</rm:rmap>

This rmap assumes that all the Eclipse projects of your Xtext project are in the same base directory; if this is not the case, because you split them, for instance, in plugins, features, doc, etc. directories, then you need to tweak this rmap accordingly, for instance, again, taken from Xsemantics,

<?xml version="1.0" encoding="UTF-8"?>
<rm:rmap xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0"
	xmlns:rm="http://www.eclipse.org/buckminster/RMap-1.0">
	<rm:property key="projects.location" value="${workspace.root}" />

	<rm:locator pattern="^it.xsemantics(?:\..+)?$"
		searchPathRef="project" />

	<rm:searchPath name="project">
		<rm:provider componentTypes="eclipse.feature,buckminster" readerType="local"
			mutable="false">
			<rm:uri format="{0}/releng/{1}">
				<bc:propertyRef key="workspace.root" />
				<bc:propertyRef key="buckminster.component" />
			</rm:uri>
		</rm:provider>
		<rm:provider componentTypes="eclipse.feature" readerType="local"
			mutable="false">
			<rm:uri format="{0}/features/{1}">
				<bc:propertyRef key="workspace.root" />
				<bc:propertyRef key="buckminster.component" />
			</rm:uri>
		</rm:provider>
		<rm:provider componentTypes="osgi.bundle" readerType="local"
			mutable="false">
			<rm:uri format="{0}/plugins/{1}">
				<bc:propertyRef key="workspace.root" />
				<bc:propertyRef key="buckminster.component" />
			</rm:uri>
		</rm:provider>
		<rm:provider componentTypes="osgi.bundle" readerType="local"
			mutable="false">
			<rm:uri format="{0}/doc/{1}">
				<bc:propertyRef key="workspace.root" />
				<bc:propertyRef key="buckminster.component" />
			</rm:uri>
		</rm:provider>
		<rm:provider componentTypes="osgi.bundle" readerType="local"
			mutable="false">
			<rm:uri format="{0}/examples/{1}">
				<bc:propertyRef key="workspace.root" />
				<bc:propertyRef key="buckminster.component" />
			</rm:uri>
		</rm:provider>
		<rm:provider componentTypes="osgi.bundle" readerType="local"
			mutable="false">
			<rm:uri format="{0}/tests/{1}">
				<bc:propertyRef key="workspace.root" />
				<bc:propertyRef key="buckminster.component" />
			</rm:uri>
		</rm:provider>
		<rm:provider componentTypes="eclipse.feature,osgi.bundle,buckminster"
			readerType="local" mutable="false">
			<rm:property key="buckminster.mutable" value="false" />
			<rm:uri format="{0}/{1}">
				<bc:propertyRef key="projects.location" />
				<bc:propertyRef key="buckminster.component" />
			</rm:uri>
		</rm:provider>
	</rm:searchPath>
</rm:rmap>

Then, you have to Component Query files (CQUERY): projects-platform.cquery, to materialize the target platform, and project.cquery, to materialize your projects in the workspace. The first one could also be used in the IDE to actually materialize the target platform in your workspace (instead of using a target definition file); the second one is useful when building headless.

projects-platform.cquery:

<?xml version="1.0" encoding="UTF-8"?>
<cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="projects-platform.rmap">
    <cq:rootRequest name="org.xtext.example.hellobuck.buckminster" componentType="buckminster"/>
    <cq:property key="target.arch" value="*"/>
    <cq:property key="target.os" value="*"/>
    <cq:property key="target.ws" value="*"/>
    <cq:advisorNode namePattern="^.*\.source$" skipComponent="true"/>
</cq:componentQuery>

Note that this cquery has an advisor node to skip .source components: thus, the materialized target platform will not contain the sources for all the features. This is useful headlessly to reduce the time to materialize the target platform. But if you use it also for materializing the target platform in the IDE, then sources are useful to inspect Java classes, and you might want to remove this advisor node (or simply create another cquery only to be used in the IDE, without that advisor node). This query refers to projects-platform.rmap.

project.cquery:

<?xml version="1.0" encoding="UTF-8"?>
<cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="project.rmap">
    <cq:rootRequest name="org.xtext.example.hellobuck.site" componentType="eclipse.feature"/>
</cq:componentQuery>

As it usually happens when using Buckminster, the cquery for materializing the projects in the (headless) workspace refers to a single feature project which, transitively refers to all the bundles and additional features of your application. Usually, this is the feature project for building the p2 repository, like in this case.

“But wait! the site feature project does not refer to test projects, since I do not want to include them in my p2 repository! So, how can Buckminster run my tests headlessly if they are not materialized?”

If you look at the feature project .site, you will note that it contains a Component Extension (CSPEX) which allows to extend the automatically inferred Component Specification for the feature project with additional specifications. In this case the buckminster.cspex contains

<?xml version="1.0" encoding="UTF-8"?>
<cs:cspecExtension xmlns:com="http://www.eclipse.org/buckminster/Common-1.0" xmlns:cs="http://www.eclipse.org/buckminster/CSpec-1.0">
	<cs:dependencies>
		<cs:dependency name="org.xtext.example.hellobuck.tests" componentType="osgi.bundle" />
	</cs:dependencies>
</cs:cspecExtension>

You see that this specification adds a dependency to the org.xtext.example.hellobuck.tests project, so that, when Buckminster materializes org.xtext.example.hellobuck.site, and its dependencies, it will also materialize org.xtext.example.hellobuck.tests. You can use this technique to materialize additional projects; for instance, in from Xsemantics, I have a main feature for tests, and a tests project for each example, thus buckminster.cspex reads as follows

<?xml version="1.0" encoding="UTF-8"?>
<cs:cspecExtension xmlns:com="http://www.eclipse.org/buckminster/Common-1.0" xmlns:cs="http://www.eclipse.org/buckminster/CSpec-1.0">
	<cs:dependencies>
		<cs:dependency name="it.xsemantics.example.fj.tests" componentType="osgi.bundle" />
		<cs:dependency name="it.xsemantics.example.expressions.tests" componentType="osgi.bundle" />
		<cs:dependency name="it.xsemantics.example.lambda.tests" componentType="osgi.bundle" />
		<cs:dependency name="it.xsemantics.tests.feature" componentType="eclipse.feature" />
	</cs:dependencies>
</cs:cspecExtension>

Finally, in the .site project, you will also find the feature.xml

<?xml version="1.0" encoding="UTF-8"?>
<feature id="org.xtext.example.hellobuck.site"
	label="org.xtext.example.hellobuck.site Feature"
	version="1.0.0.qualifier">
	<includes
		id="org.xtext.example.hellobuck.sdk"
		version="0.0.0"/>
</feature>

which includes the .sdk feature of your project (Remember: when building the site.p2 on a feature project with Buckminster, you will build a p2 repository NOT for the very feature, but for the included features), and the category.xml to give categories to your features in the p2 repository:

<?xml version="1.0" encoding="UTF-8"?>
<site>
	<feature id="org.xtext.example.hellobuck.sdk" version="0.0.0">
		<category name="main"/>
	</feature>
	<feature id="org.xtext.example.hellobuck.sdk.source" version="0.0.0">
		<category name="main.source"/>
	</feature>
   <category-def name="main" label="Hellobuck"/>
   <category-def name="main.source" label="Source for Hellobuck"/>
</site>

Note that the .sdk.source feature is not a “real” feature project in your workspace, but, by default, Buckminster will automatically build a source jar for all your features and bundles.

That’s all! Many thanks to the author of this wizard, Dennis Hübner, for making it and for many suggestions!

Have fun with Bucky and Xtext :)

 

Be Sociable, Share!