Fornax Workflow Plugin 3.4.0 released

Finally I have managed to release a new version of the Fornax Maven Workflow Plugin again. This Maven plugin allows execution of MWE and MWE2 workflows from Maven, which is typically the case if you want to automate the build of Xtext or Xpand based projects with Maven.

The new version has some interesting new features which may be worth upgrading:

  • m2e lifecycle metadata
  • workflow dependencies can be configured as plugin dependencies
  • more options to detect changes that trigger the build
  • force/skip parameter to configure workflow execution overrides per configuration
  • log level detection when running in forked JVM

I have especially to thank Dieter Schulten from escalon, who has been willing to contribute to this release. He has mainly contributed the m2e configuration and some tests for the plugin.

Unofficial release 2.2.2 of the Xtend Maven Plugin

In my previous post I showed how to use the maven-xtend-plugin to compile Xtend sources within a Maven build. This worked fine in simple cases, but there was a limitation of only one supported source directory (the one configured by build/sourceDirectory property), see Bug#367914. Unfortunately already each Xtext project has already 2 source folders (src and src-gen), so this made the plugin unusable for Xtext projects and forced checking in the generated Java files. This causes terrible problems with version control, especially when working on a team.

In the meantime Bug#367914 was resolved, but only for the upcoming Xtend 2.3 (Eclipse Juno) version. This urgent bugfix was not available on a Maven repository. But I needed it for project Spray to finally get rid of the merge conflicts and delete the generated sources from the repository (issue#94). Spray is still using Xtext/Xtend version 2.2.1 and depends on public available Maven artifacts. Since the bugfix won’t be available from the Xtend project for the 2.2.1 release and Eclipse Juno is still some way to go I decided to backport the current 2.3.0 M6 sources of the Xtend Maven Plugin to 2.2.x and create an unofficial version 2.2.2. This plugin I have deployed to the Fornax Repository to make it public available.

For an usage example refer to the POMs of project Spray: pom.xml, parent pom.xml.

Xtext 2.2 finally brings Maven support for Xtend

I cannot tell how often I was asked since introduction of Xtend2 how to compile them in a Maven build. This was just not possible until now due to the problem that to load and compile an Xtend class it is necessary to compile all of the Java classes that the Xtend class on before, and Java classes might depend on Xtend classes to be translated in order to be compilable.

Xtext 2.2.0 was just released yesterday, and for me the most important new feature is direct support through the xtend-maven-plugin plugin. I could not hesitate to test this feature, and created a small example.

Scenario

I have set up a small Maven project with 2 Java classes and 1 Xtend class that depend on each other.

To translate XtendClass1 to Java code and compile it requires that JavaClass1 is compiled before, and to compile JavaClass2 it is necessary that XtendClass is translated to Java code before.

Maven POM Configuration

I have decided to not use Maven Tycho for this example, and use the typical structure of an Eclipse project with usage of Xtend, i.e. sources in folder /src (instead /src/main/java) and xtend-gen to generate the Java code for Xtend classes to.

Repositories

Since shortly Xtext artifacts are available on maven.eclipse.org. The new Maven plugin is available on http://build.eclipse.org/common/xtend/maven/

	<repositories>
		<repository>
			<id>maven.eclipse.org</id>
			<url>http://maven.eclipse.org/nexus/content/groups/public/</url>
		</repository>
		<repository>
			<id>xtend</id>
			<url>http://build.eclipse.org/common/xtend/maven/</url>
		</repository>
	</repositories>
	<pluginRepositories>
		<pluginRepository>
			<id>xtend</id>
			<url>http://build.eclipse.org/common/xtend/maven/</url>
		</pluginRepository>
		<pluginRepository>
			<id>fornax</id>
			<url>http://fornax-platform.org/nexus/content/groups/public/</url>
		</pluginRepository>
	</pluginRepositories>

Dependencies

This small project setup requires just a mimimal set of dependencies to be configured:

	<dependencies>
		<dependency>
			<groupId>org.eclipse.xtend2</groupId>
			<artifactId>org.eclipse.xtend2.lib</artifactId>
			<version>2.2.0</version>
		</dependency>
		<dependency>
			<groupId>org.eclipse.xtext</groupId>
			<artifactId>org.eclipse.xtext.xtend2.lib</artifactId>
			<version>2.2.0.v201112061305</version>
		</dependency>
		<dependency>
			<groupId>com.google.inject</groupId>
			<artifactId>com.google.inject</artifactId>
			<version>2.0.0.v201105231817</version>
		</dependency>
	</dependencies>

Source folders

The main source folder src/ can be configured with the build/sourceDirectory setting, for xtend-gen we need the support of the build-helper-maven-plugin.

	<build>
		<sourceDirectory>src</sourceDirectory>
		<plugins>
			<plugin>
				<groupId>org.codehaus.mojo</groupId>
				<artifactId>build-helper-maven-plugin</artifactId>
				<version>1.7</version>
				<executions>
					<execution>
						<id>add-source</id>
						<phase>generate-sources</phase>
						<goals>
							<goal>add-source</goal>
						</goals>
						<configuration>
							<sources>
								<source>xtend-gen</source>
							</sources>
						</configuration>
					</execution>
				</executions>
			</plugin>
			...

The xtend-gen folder must be emptied when executing mvn clean, this requires some additional configuration of the maven-clean-plugin.

			<plugin>
				<artifactId>maven-clean-plugin</artifactId>
				<version>2.4.1</version>
				<configuration>
					<filesets>
						<fileset>
							<directory>xtend-gen</directory>
							<includes>
								<include>**</include>
							</includes>
						</fileset>
					</filesets>
				</configuration>
			</plugin>

xtend-maven-plugin

Now let’s finally come to the Xtend plugin. The plugin is configured as follows:

			<plugin>
				<groupId>org.eclipse.xtend2</groupId>
				<artifactId>xtend-maven-plugin</artifactId>
				<version>2.2.2</version>
				<executions>
					<execution>
						<goals>
							<goal>compile</goal>
							<goal>testCompile</goal>
						</goals>
						<configuration>
							<outputDirectory>xtend-gen</outputDirectory>
						</configuration>
					</execution>
				</executions>
			</plugin>

The outputDirectory is optional, but leaving it out the sources are generated to src/main/xtend-gen, and we want to generate to xtend-gen in the project root.

The plugin does not define a lifecycle mapping for M2E, which leads to an error marker “Plugin execution not covered by lifecycle configuration”. This is a well known issue when using M2E, and requires some additional work on the plugin. I have opened Bug#366118 for this.

Sample Project

The project described above is shared on Github: https://github.com/kthoms/xtext-experimental

In Eclipse the project looks like this after import (assuming the M2E plugin installed):

Build Execution

The classes in the project can now be built using mvn clean install. This will produce the following output:

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building xtend-maven-classic 2.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ xtend-maven-classic ---
[INFO] Deleting /Users/thoms/git/xtext-experimental/maven/xtend.maven/target
[INFO] Deleting /Users/thoms/git/xtext-experimental/maven/xtend.maven/xtend-gen (includes = [**], excludes = [])
[INFO] 
[INFO] --- build-helper-maven-plugin:1.7:add-source (add-source) @ xtend-maven-classic ---
[INFO] Source directory: /Users/thoms/git/xtext-experimental/maven/xtend.maven/xtend-gen added.
[INFO] 
[INFO] --- xtend-maven-plugin:2.2.0:compile (default) @ xtend-maven-classic ---
[WARNING] 
WARNING: 	XtendClass1.xtend - /Users/thoms/git/xtext-experimental/maven/xtend.maven/src/mypackage/XtendClass1.xtend
7: The value of the field XtendClass1.cls is not used
[INFO] Compiling 1 source file to xtend-gen
[INFO] 
[INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ xtend-maven-classic ---
[WARNING] Using platform encoding (MacRoman actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/thoms/git/xtext-experimental/maven/xtend.maven/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ xtend-maven-classic ---
[WARNING] File encoding has not been set, using platform encoding MacRoman, i.e. build is platform dependent!
[INFO] Compiling 3 source files to /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.923s
[INFO] Finished at: Thu Dec 08 23:01:42 CET 2011
[INFO] Final Memory: 20M/81M
[INFO] ------------------------------------------------------------------------

From the output it can be seen that at the end 3 Java classes are compiled, after the xtend-maven-plugin created the Java code for the Xtend class.

Enabling debug output with -X reveals some more insight, how the plugin works:

[DEBUG] Configuring mojo 'org.eclipse.xtend2:xtend-maven-plugin:2.2.0:compile' with basic configurator -->
[DEBUG]   (f) outputDirectory = xtend-gen
[DEBUG]   (f) project = MavenProject: org.eclipse.xtext.example:xtend-maven-classic:2.2.0-SNAPSHOT @ /Users/thoms/git/xtext-experimental/maven/xtend.maven/pom.xml
[DEBUG]   (f) tempDirectory = /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend
[DEBUG] -- end configuration --
[DEBUG] load xtend file 'file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/mypackage/XtendClass1.xtend'
[DEBUG] Parsing took: 42 ms
...
[DEBUG] create java stub 'mypackage/XtendClass1.java'
[DEBUG] invoke batch compiler with '-cp /Users/thoms/git/xtext-experimental/maven/xtend.maven/src:/Users/thoms/.m2/repository/org/eclipse/xtend2/org.eclipse.xtend2.lib/2.2.0/org.eclipse.xtend2.lib-2.2.0.jar:/Users/thoms/.m2/repository/com/google/guava/guava/10.0.1/guava-10.0.1.jar:/Users/thoms/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xtend2.lib/2.2.0.v201112061305/org.eclipse.xtext.xtend2.lib-2.2.0.v201112061305.jar:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xbase.lib/2.2.0.v201112061305/org.eclipse.xtext.xbase.lib-2.2.0.v201112061305.jar:/Users/thoms/.m2/repository/com/google/inject/com.google.inject/2.0.0.v201105231817/com.google.inject-2.0.0.v201105231817.jar -d /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/classes -1.5 -proceedOnError /Users/thoms/git/xtext-experimental/maven/xtend.maven/src /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/stubs'
[DEBUG] classpath used for Xtend compilation : [file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/, file:/Users/thoms/.m2/repository/org/eclipse/xtend2/org.eclipse.xtend2.lib/2.2.0/org.eclipse.xtend2.lib-2.2.0.jar, file:/Users/thoms/.m2/repository/com/google/guava/guava/10.0.1/guava-10.0.1.jar, file:/Users/thoms/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar, file:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xtend2.lib/2.2.0.v201112061305/org.eclipse.xtext.xtend2.lib-2.2.0.v201112061305.jar, file:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xbase.lib/2.2.0.v201112061305/org.eclipse.xtext.xbase.lib-2.2.0.v201112061305.jar, file:/Users/thoms/.m2/repository/com/google/inject/com.google.inject/2.0.0.v201105231817/com.google.inject-2.0.0.v201105231817.jar, file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/, file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/classes/]

The plugin invokes the Xtend Batch compiler through a spawned JVM. The batch compiler is realized in class org.eclipse.xtext.xtend2.compiler.batch.Xtend2BatchCompiler from the newly added plugin org.eclipse.xtext.xtend2.standalone.

After the build XtendClass1.java is generated as expected to xtend-gen. When opening the target folder it can be seen that the plugin produces a Java stub class to target/stubs for the Xtend class. Finally, everything is compiled and the project is clean :-)

Limitation

The plugin only takes one source folder into account, namely the one configured by build/sourceDirectory. This is a problem when you have multiple source folders, which is quite typical for Xtext projects, namely src and src-gen. This is reported as , which is fixed in the meantime. I have deployed a patched version as unofficial release 2.2.2.

Conclusion

I deeply desired this plugin and this finally allows that the xtend-gen folder does not need to be checked in. The sample used here was just simple. Next would be a real life project were I want to apply the plugin, most likely in Project Spray. This project uses Xtend based code generation heavily and Maven Tycho for the build. The project has not been upgraded to Xtext 2.2.0 (of course, it was just released), and I guess we have to upgrade to Xtext 2.2.0 before we can use the plugin. But if this allows us to finally remove the xtend-gen folders from the repository this alone is worth upgrading.

Nice work, Xtext team!

Xtext 2.1.1 artifacts available on Eclipse Maven repository

Until now it was always a struggle to make Xtext (or Eclipse) artifacts available on a public available repository. It seems that this has now finally found a good end.

Today Achim Demelt pointed me to a Google+ posting from Dennis Huebner (our Xtext build manager) where he announced that Xtext artifacts are now automatically available on the Nexus repository at maven.eclipse.org.

The given POMs have all dependencies from their bundle manifests. This is the recommended “Maven way”, but you may face that too many dependencies are pulled transitive, and you may have to cut them down by explicit exclusions. I haven’t tested this with the new repository, but this is what I experienced when doing a similar approach earlier with Xtext 2.0.1, whose artifacts are deployed on the Fornax Repository. Use the dependency:tree plugin, or better M2E, to identify possible exclusions.

Xtext, Maven and Source Control

Although I more and more use Git for source control, I face Subversion still frequently in daily life. I follow the rule that I don’t want to check in what can be generated in a headless build, where I usually use Maven. Xtext projects have the source folders src-gen and xtend-gen, where generated sources go to. Further, you will usually have 3 projects: The grammar project, the UI project and a test project.

In this article I will show some guidelines that I follow when checking in Xtext projects into source control.

Handling the src-gen folder

The content of src-gen can be produced during a build, so I do not check in the content of it. What needs to be considered is that is no good idea to ignore the folder completely, since it would be missing when fresh checking out the project. This leads to an incomplete build path, and the MWE2 workflow is not executable from Eclipse until the folder was created. This is because the .mwe2 file needs to be on the project’s classpath, which is not the folder src, but the output folder, bin or target/classes. It will be copied automatically with default workspace settings only when the build path is complete. This behavior can be changed in the workspace / project settings in Java / Compiler / Building.

If you rate down build path problems to warning level or just disable build abortion on errors, the .mwe2 file will be copied anyway to the target folder. If using project specific settings you could even check in these settings and share it in the team.

I prefer to check in the src-gen folder, but avoid that any content gets checked in. To do so, add the folder with svn add, but none of its content! Best you do this immediately after the Xtext project was created. After the folder was scheduled for adding, you can set the svn:ignore property to value “*”.

The xtend-gen folder

At the moment it is not possible to compile .xtend files during a Maven build, so the content of xtend-gen must be checked in and handled like handcrafted sources. To cut this story short: Xtend and Java files depend usually on each other. Xtend files cannot be compiled until Java sources from the same project were compiled, and some Java sources will depend on about-to-be-compiled Xtend sources. So the Java source compilation fails, aborting the build.

I hope to solve this problem soon.

Some ignore patterns

On the root of each of the 3 Xtext projects, set the property svn:ignore to this value:

bin
target
.DS_Store
plugin.xml_gen

Configure Maven clean plugin

When running mvn clean on your project you would like to to clean up all derived resources. The target folder will be removed by default, but this is not enough. The content of the src-gen folders must be cleaned up as well. This can be done by configuring the maven-clean-plugin. One thing to consider is that Xtext has 3 projects. It is not correct to configure the clean plugin for all 3 projects due the lifecycle of a Maven build: The projects will be build after each other, executing all lifecycles per project. If you execute a “mvn clean install” this would have the effect that

  1. clean is called on the grammar project
  2. in phase generate-sources the Xtext implementation classes are produced to all 3 projects
  3. the sources for the grammar project are compiled
  4. now the build continues with the next module, e.g. the UI project
  5. the project will be cleaned, removing the previously generated sources
  6. compilation fails, since the generated sources are missing now

For this reason the clean plugin must be configured only in the grammar project, and it should clean up the sources from all 3 projects. Now comes again a detail related to source control: Since we want to have the src-gen checked in, we have to avoid that the SCM related folders and files are removed. Otherwise this would confuse your SCM system. So these files must be excluded from deletion. This leads to the following configuration:

<plugin>
  <artifactId>maven-clean-plugin</artifactId>
  <configuration>
    <filesets>
      <fileset>
        <directory>src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
      <fileset>
        <directory>../${project.artifactId}.ui/src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
      <fileset>
        <directory>../${project.artifactId}.tests/src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
    </filesets>
  </configuration>
</plugin>

Xtext 2.0 / Xpand 1.1 plugins available as Maven artifacts

It’s now one week after release of Eclipse Indigo, and with that Xtext 2.0 and Xpand 1.1. For those who use Maven without Tycho to build projects using these frameworks it is of course necessary to have all related plugins available in some Maven repository.

With the help of the osgi-to-maven2 project I was now able with some tweaks to create the necessary POMs and deploy scripts for the plugins. What osgi-to-maven2 basically does is to analyze the plugin manifests and create POMs for them including dependencies. The project is not maintained at the moment and has a problem especially with the parsing of the plugins org.eclipse.ui.views and org.eclipse.ui.workbench, which have some strange directives in the Export-Package section where I could not find documentation about. The underlying manifest parser from Springsource (com.springsource.util.osgi) vomits when parsing those manifests, and also upgrading to the latest release does not help. Time to open a bug.


Export-Package: org.eclipse.ui;mandatory:="ui.workb
ench",org.eclipse.ui.about,org.eclipse.ui.actions;ui.workbench=split;
mandatory:="ui.workbench",org.eclipse.ui.activities,org.eclipse.ui.ap
plication,org.eclipse.ui.branding,org.eclipse.ui.browser;ui.workbench
=split
;mandatory:="ui.workbench",org.eclipse.ui.commands,org.eclipse.

Also the osgi-to-maven2 tool does not handle absent optional libraries so far.

The Xtext 2.0, Xpand 1.1 and dependent plugins are available at the Fornax Platform’s repository http://fornax-platform.org/nexus/content/groups/public/ under the groupId org.eclipse.plugins.

Based on the deployed plugins it should be possible to setup a plain Maven build for Xtext 2 based projects.

Using the JDT compiler in normal Maven builds

The Maven Compiler Plugin can be customized to use a different compiler than Javac. One of the options listed for the compilerId setting is “eclipse“. To use it you will have to add a dependency to the plexus-compiler-eclipse (current version: 1.8.1).

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-compiler-plugin</artifactId>
	<configuration>
		<compilerId>eclipse</compilerId>
		<source>1.5</source>
		<target>1.5</target>
		<optimize>true</optimize>
	</configuration>
	<dependencies>
		<dependency>
			<groupId>org.codehaus.plexus</groupId>
			<artifactId>plexus-compiler-eclipse</artifactId>
			<version>1.8.1</version>
			<scope>runtime</scope>
		</dependency>
	</dependencies>
</plugin>

A pitfall here is that the compiler compliance level is not automatically set through the target setting. You have to set also optimize=true. Otherwise Java 1.5 sources won’t compile.

The plexus-compiler-eclipse (version 1.8.1) takes a rather old version of JDT, 3.3.0-v_771. To use another version you will have to deploy a newer version of JDT yourself into your repository and list this dependency before plexus-compiler-eclipse.

Tycho build works also for oAW 4.3 projects

Yesterday I showed how Xtext 1.0 projects can be build with Maven Tycho. So why shouldn’t the same setup don’t work for ‘old’ openArchitectureWare Xpand projects? This works like a charm. I took the simple project that the Xpand project wizard creates and added the POM, and it worked! You can download the sample project here. See the POM below, a setup for other oAW Xpand projects should be rather similar.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>p2.osgi.bundle</groupId>
	<artifactId>org.openarchitectureware.example.xpand.tycho</artifactId>
	<version>1.0.0</version>
	<packaging>eclipse-plugin</packaging>
	<name>org.openarchitectureware.example.xpand.tycho</name>

	<properties>
		<version.tycho>0.9.0</version.tycho>
	</properties>

	<build>
		<resources>
			<resource>
				<directory>src</directory>
			</resource>
		</resources>
		<plugins>
			<plugin>
				<groupId>org.sonatype.tycho</groupId>
				<artifactId>tycho-maven-plugin</artifactId>
				<version>${version.tycho}</version>
				<extensions>true</extensions>
			</plugin>
			<plugin>
				<groupId>org.sonatype.tycho</groupId>
				<artifactId>target-platform-configuration</artifactId>
				<version>${version.tycho}</version>
				<configuration>
					<resolver>p2</resolver>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.fornax.toolsupport</groupId>
				<artifactId>fornax-oaw-m2-plugin</artifactId>
				<version>3.1.0-SNAPSHOT</version>
				<configuration>
					<workflowEngine>oaw</workflowEngine>
					<workflowDescriptor>workflow/generator.oaw</workflowDescriptor>
				</configuration>
				<executions>
					<execution>
						<phase>generate-sources</phase>
						<goals>
							<goal>run-workflow</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<artifactId>maven-clean-plugin</artifactId>
				<version>2.4.1</version>
				<configuration>
					<filesets>
						<fileset>
							<directory>src-gen</directory>
							<includes>
								<include>**/**</include>
							</includes>
						</fileset>
					</filesets>
				</configuration>
			</plugin>
		</plugins>
	</build>
	<repositories>

		<repository>
			<id>p2.openarchitectureware</id>
			<url>http://www.openarchitectureware.org/updatesite/milestone/</url>
			<layout>p2</layout>
		</repository>
		<repository>
			<id>p2.eclipse.helios</id>
			<url>http://download.eclipse.org/releases/helios</url>
			<layout>p2</layout>
		</repository>
		<!-- At the moment the Fornax plugin is only available as snapshot -->
		<repository>
			<id>fornax-snapshots</id>
			<url>http://www.fornax-platform.org/archiva/repository/snapshots/</url>
			<releases>
				<enabled>false</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</repository>
	</repositories>
	<pluginRepositories>
		<pluginRepository>
			<id>fornax-snapshots</id>
			<url>http://www.fornax-platform.org/archiva/repository/snapshots/</url>
			<releases>
				<enabled>false</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</pluginRepository>
	</pluginRepositories>
</project>

Building Xtext projects with Maven Tycho

Last year I showed how Xtext 0.7.2 projects can be build with Maven. It requires the installation of all related plugins as Maven artifacts in a Maven2 repository, which is a rather hard and error prone task. I was asked many times since then whether I would install Xtext 1.0 artifacts or even milestone versions into the openArchitectureWare Maven2 repository. Although I can really understand this requirement my hope was that I could avoid this and show a way to enable the build with Maven Tycho. Now the time has come that Tycho is mature enough and finally I had the time to do the necessary extension for the Fornax Maven Workflow Plugin to support the recently added MWE2 workflow engine, which is used from Xtext now as default. Maven Tycho and the Fornax plugin will allow to build Xtext projects in the most natural way possible.

A basic requirement for software builds is that everything can be build just from the sources. Also it is a good practice that no generated sources are checked in into a source repository. Unfortunately this often does not hold for Xtext projects, since it is a quite hard task to execute the Xtext generator in a build process. Yes, it is possible, but until now users often have to solve it again and again. Search the net, most projects seen there will just check in all the sources. The Fornax plugin and Maven will standardize this behavior.

What do you need to enable Xtext DSL builds with Tycho? Not much. Basically a Maven 3 installation (the beta is enough) and access to the internet to access the public repositories. A better idea is to use a repository manager in the intranet (like Nexus) to act as proxy for the public repositories. You will have to add some POMs to your projects. That’s it.

I’ll show now how to enable the basic project that you get from the Xtext project wizard get build with Maven Tycho.

1. Install Maven 3

Download Maven 3 from Apache, unpack it and put the bin folder on your path. Check the installation by typing ‘mvn –version‘ in a shell. You should something like this:

mvn --version
Apache Maven 3.0-beta-1 (r935667; 2010-04-19 19:00:39+0200)
Java version: 1.6.0_20
Java home: /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
Default locale: de_DE, platform encoding: MacRoman
OS name: "mac os x" version: "10.5.8" arch: "x86_64" Family: "mac"

2. Create the Xtext project

Select in Eclipse File / New / Project / Xtext Project and leave all the defaults. The project name should be org.xtext.example.mydsl for this tutorial. The projects will be created and almost empty for now. The aim is to get everything build by just adding the POMs.

3. Adjust plugin manifests

With the default Manifests the build does not execute successfully. The build fails because it has problems to resolve the logging infrastructure. It is sufficient to add Import-Package entries to resolve SLF4J and Logback. Open the plugin Manifest of the org.xtext.example.mydsl project, go to the Dependencies page and add the following Imported Packages:

  • ch.qos.logback.classic
  • ch.qos.logback.core.joran.spi
  • org.slf4j

For each of them open the Properties dialog and set the “optional” flag. Otherwise you could get problems when deploying these plugins. (Thanks to Christian Amman for this tip!)

Open the Manifest of org.xtext.example.mydsl.generator, add the same packages and additionally

  • org.apache.commons.logging

Again, set the imported packages to optional.

4. Add POM files

Each project will get a POM file. A Parent POM will contain the common settings and will act as a reactor POM also. Normally the Parent POM is placed in the directory above the modules, or here the plugin projects. This would be the workspace root. In this example we will place the Parent POM in the DSL grammar project. To make builds easier this POM will have the default name pom.xml. On the other side we have to add also a POM for the grammar project itself. Since it will be in the same directory we have to find another name, and we will call it pom-grammar.xml. For the other projects (.generator and .ui) we will stay with pom.xml.

4.1 org.xtext.example.mydsl/pom.xml

The Parent POM configures the Tycho plugins and aggregates the grammar project, UI project and the generator. Also the required repositories are configured here. Note that the folder “src” is added as resource path, this will add this folder to the classpath. The workflow execution would fail if the workflow module cannot be found on the classpath.

<!--?xml version="1.0" encoding="UTF-8"?-->
	4.0.0
	p2.osgi.bundle
	org.xtext.example.mydsl.parent
	1.0.0
	pom
	org.xtext.example.mydsl.MyDsl - Parent

		0.9.0

		./pom-grammar.xml
		../org.xtext.example.mydsl.ui
		../org.xtext.example.mydsl.generator

		<!-- The src directory must be named as resource dir to put it on the build classpath. This is required to resolve the 			workflow module named in the .mwe2 file -->

			src

				org.sonatype.tycho
				tycho-maven-plugin
				${version.tycho}
				true

				org.sonatype.tycho
				target-platform-configuration
				${version.tycho}

					p2

					org.fornax.toolsupport
					fornax-oaw-m2-plugin
					3.1.0-SNAPSHOT

						mwe2

							generate-sources
								run-workflow

			p2.eclipse.helios

http://download.eclipse.org/releases/helios

			p2

		<!-- At the moment the Fornax plugin is only available as snapshot -->

			fornax-snapshots

http://www.fornax-platform.org/archiva/repository/snapshots/

				false

				true

org.xtext.example.mydsl/pom-grammar.xml

The build of the grammar project needs to execute the workflow src/org/xtext/example/mydsl/GenerateMyDsl.mwe2. All other settings are already managed by the parent.

<!--?xml version="1.0" encoding="UTF-8"?-->
	4.0.0

		p2.osgi.bundle
		org.xtext.example.mydsl.parent
		1.0.0
		../org.xtext.example.mydsl/pom.xml

	org.xtext.example.mydsl
	eclipse-plugin
	org.xtext.example.mydsl.MyDsl - Grammar

				org.fornax.toolsupport
				fornax-oaw-m2-plugin

					src/org/xtext/example/mydsl/GenerateMyDsl.mwe2

org.xtext.example.mydsl.ui/pom.xml

The build configuration of the UI plugin is even easier. Only the reference to the parent and the minimum project coordinates need to be configured.

<!--?xml version="1.0" encoding="UTF-8"?-->
	4.0.0

		p2.osgi.bundle
		org.xtext.example.mydsl.parent
		1.0.0
		../org.xtext.example.mydsl/pom.xml

	org.xtext.example.mydsl.ui
	eclipse-plugin
	org.xtext.example.mydsl.MyDsl - UI

org.xtext.example.mydsl.generator/pom.xml

The generator plugin must execute its workflow. Note that the src directory and the compile directory of the grammar project are set as resource directory.

<!--?xml version="1.0" encoding="UTF-8"?-->
	4.0.0

		p2.osgi.bundle
		org.xtext.example.mydsl.parent
		1.0.0
		../org.xtext.example.mydsl/pom.xml

	org.xtext.example.mydsl.generator
	eclipse-plugin
	org.xtext.example.mydsl.MyDsl - Generator

			src
			<!-- Add the grammar classes to the classpath -->
			../org.xtext.example.mydsl/target/classes

				org.fornax.toolsupport
				fornax-oaw-m2-plugin

					mwe2
					src/workflow/MyDslGenerator.mwe2

5. Execute the build

Now the build should be executable. Run ‘mvn clean install‘ in the root of the org.xtext.example.mydsl project. Here is an excerpt of the console log you should get. The complete log can be viewed here.

mvn install
[INFO] Scanning for projects...
[WARNING] No explicit target runtime environment configuration. Build is platform dependent.
[INFO] Resolving target platform for project MavenProject: p2.osgi.bundle:org.xtext.example.mydsl:1.0.0 @ /Users/thoms/temp/tycho/trunk/trunk/org.xtext.example.mydsl/pom-grammar.xml
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
[INFO] Adding repository http://download.eclipse.org/releases/helios
[INFO] Adding repository http://download.eclipse.org/releases/helios
[WARNING] No explicit target runtime environment configuration. Build is platform dependent.
[INFO] Resolving target platform for project MavenProject: p2.osgi.bundle:org.xtext.example.mydsl.ui:1.0.0 @ /Users/thoms/temp/tycho/trunk/trunk/org.xtext.example.mydsl.ui/pom.xml
[INFO] Adding repository (cached) http://download.eclipse.org/releases/helios
[WARNING] No explicit target runtime environment configuration. Build is platform dependent.
[INFO] Resolving target platform for project MavenProject: p2.osgi.bundle:org.xtext.example.mydsl.generator:1.0.0 @ /Users/thoms/temp/tycho/trunk/trunk/org.xtext.example.mydsl.generator/pom.xml
[INFO] Adding repository (cached) http://download.eclipse.org/releases/helios
[WARNING] No explicit target runtime environment configuration. Build is platform dependent.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] org.xtext.example.mydsl.MyDsl - Parent
[INFO] org.xtext.example.mydsl.MyDsl - Grammar
[INFO] org.xtext.example.mydsl.MyDsl - UI
[INFO] org.xtext.example.mydsl.MyDsl - Generator
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building org.xtext.example.mydsl.MyDsl - Parent 1.0.0
[INFO] ------------------------------------------------------------------------
....
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] org.xtext.example.mydsl.MyDsl - Parent ............ SUCCESS [0.681s]
[INFO] org.xtext.example.mydsl.MyDsl - Grammar ........... SUCCESS [26.418s]
[INFO] org.xtext.example.mydsl.MyDsl - UI ................ SUCCESS [3.103s]
[INFO] org.xtext.example.mydsl.MyDsl - Generator ......... SUCCESS [10.844s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

5. Miscalleneous

5.1 ANTLR 3 download

Xtext relies on ANTLR 3, which cannot be hosted at Eclipse.org due to incompatible licenses. Therefore Xtext has introduced an automatic download (FYI: This behavior is implemented in AntlrToolFacade) that you have to confirm once:

*ATTENTION*
It is recommended to use the ANTLR 3 parser generator (BSD licence - http://www.antlr.org/license.html).
Do you agree to download it (size 1MB) from 'http://download.itemis.com/antlr-generator-3.0.1.jar'? (type 'y' or 'n' and hit enter)

Now an automatic build is not so responsive to questions like these. The plugin will confirm this download automatically.

5.2 Workflow execution problems

If the Maven plugin does not execute the MWE2 workflow properly it is not so responsive at the moment to name the root cause. But when running the Maven command with “-X” it will output the full Java command including classpath to the console. Copy the full command and try to execute it from the same directory. This might help to detect the problem you have.

/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java -classpath /Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xpand/1.0.0.v201006150611/org.eclipse.xpand-1.0.0.v201006150611.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xtend/1.0.0.v201006150611/org.eclipse.xtend-1.0.0.v201006150611.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.mwe.core/1.0.0.v201006150535/org.eclipse.emf.mwe.core-1.0.0.v201006150535.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.ecore/2.6.0.v20100614-1136/org.eclipse.emf.ecore-2.6.0.v20100614-1136.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.common/2.6.0.v20100614-1136/org.eclipse.emf.common-2.6.0.v20100614-1136.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.apache.commons.cli/1.0.0.v20080604-1500/org.apache.commons.cli-1.0.0.v20080604-1500.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.mwe2.runtime/1.0.0.v201006150446/org.eclipse.emf.mwe2.runtime-1.0.0.v201006150446.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/com.ibm.icu/4.2.1.v20100412/com.ibm.icu-4.2.1.v20100412.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xtext/1.0.0.v201006170321/org.eclipse.xtext-1.0.0.v201006170321.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.ecore.xmi/2.5.0.v20100521-1846/org.eclipse.emf.ecore.xmi-2.5.0.v20100521-1846.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xtext.util/1.0.0.v201006170321/org.eclipse.xtext.util-1.0.0.v201006170321.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/com.google.collect/0.8.0.v201006170321/com.google.collect-0.8.0.v201006170321.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/com.google.inject/2.0.0.v201003051000/com.google.inject-2.0.0.v201003051000.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.antlr.runtime/3.0.0.v200803061811/org.antlr.runtime-3.0.0.v200803061811.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.mwe.utils/1.0.0.v201006150535/org.eclipse.emf.mwe.utils-1.0.0.v201006150535.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.mwe2.launch/1.0.0.v201006150907/org.eclipse.emf.mwe2.launch-1.0.0.v201006150907.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.mwe2.language/1.0.0.v201006150907/org.eclipse.emf.mwe2.language-1.0.0.v201006150907.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xtext.common.types/1.0.0.v201006170321/org.eclipse.xtext.common.types-1.0.0.v201006170321.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xtend.typesystem.emf/1.0.0.v201006150611/org.eclipse.xtend.typesystem.emf-1.0.0.v201006150611.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/ch.qos.logback.classic/0.9.19.v20100519-1505/ch.qos.logback.classic-0.9.19.v20100519-1505.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/ch.qos.logback.core/0.9.19.v20100419-1216/ch.qos.logback.core-0.9.19.v20100419-1216.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.apache.commons.logging/1.1.1.v201005080502/org.apache.commons.logging-1.1.1.v201005080502.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.slf4j.api/1.5.11.v20100519-1910/org.slf4j.api-1.5.11.v20100519-1910.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/ch.qos.logback.slf4j/0.9.19.v20100519-1910/ch.qos.logback.slf4j-0.9.19.v20100519-1910.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.slf4j.log4j/1.5.11.v20100419-1106/org.slf4j.log4j-1.5.11.v20100419-1106.jar org.eclipse.emf.mwe2.launch.runtime.Mwe2Launcher /Users/thoms/Development/workspaces/oaw-v5-test/org.xtext.example.mydsl.generator/src/workflow/MyDslGenerator.mwe2

5.3 Fornax Plugin Snapshot version

The Maven plugin org.fornax.toolsupport:fornax-oaw-m2-plugin is not released yet. Yesterday I have added the necessary support for MWE2 and deployed a snapshot version 3.1.0-SNAPSHOT. Some minor issues have to be solved before it will be released.

5.4 Missing log output

The plugin forks a JVM with help of the Java ant task type. At the moment the output from the forked JVM does not get redirected to the Plugin’s log. This is one issue to solve for the Plugin before release. It makes debugging problems a bit harder, but when copying the executed Java command that you get from the debug output and execute yourself you get a better impression of the error.

5.5 Download sources

For your convenience you can download the example project here.

Tycho slides uploaded

Yesterday I gave a presentation about Maven 3 and Tycho for the Dortmunder Vortragsreihe, a public event series from itemis. A similar talk I gave at the JAX 2010 conference last week, but with less presentation time and thus a less content. I worked on learning about Tycho now quite a while and some bugs and the current lack of documentation led to some restless evenings in the hotel. At the end I wanted to have a continuous build for a non trivial RCP application build. As the example project I wanted to build I took Kai Tödter’s MP3Manager RCP sample project. Finally I got the project building with Tycho. There are still some issues, especially with the product export. I will use this project now to create situations that show bugs and report them to the Tycho team. They are working really hard at the moment to improve Tycho, and I’m sure that many issues will be resolved soon.

Anyway, the slides that I used yesterday were just uploaded, and I hope they will be useful to get started with Tycho.

Maven 3 / Tycho

In the future I will report more about Tycho, so stay tuned!