Fornax Workflow Plugin 3.4.0 released

Finally I have managed to release a new version of the Fornax Maven Workflow Plugin again. This Maven plugin allows execution of MWE and MWE2 workflows from Maven, which is typically the case if you want to automate the build of Xtext or Xpand based projects with Maven.

The new version has some interesting new features which may be worth upgrading:

  • m2e lifecycle metadata
  • workflow dependencies can be configured as plugin dependencies
  • more options to detect changes that trigger the build
  • force/skip parameter to configure workflow execution overrides per configuration
  • log level detection when running in forked JVM

I have especially to thank Dieter Schulten from escalon, who has been willing to contribute to this release. He has mainly contributed the m2e configuration and some tests for the plugin.

Unofficial release 2.2.2 of the Xtend Maven Plugin

In my previous post I showed how to use the maven-xtend-plugin to compile Xtend sources within a Maven build. This worked fine in simple cases, but there was a limitation of only one supported source directory (the one configured by build/sourceDirectory property), see Bug#367914. Unfortunately already each Xtext project has already 2 source folders (src and src-gen), so this made the plugin unusable for Xtext projects and forced checking in the generated Java files. This causes terrible problems with version control, especially when working on a team.

In the meantime Bug#367914 was resolved, but only for the upcoming Xtend 2.3 (Eclipse Juno) version. This urgent bugfix was not available on a Maven repository. But I needed it for project Spray to finally get rid of the merge conflicts and delete the generated sources from the repository (issue#94). Spray is still using Xtext/Xtend version 2.2.1 and depends on public available Maven artifacts. Since the bugfix won’t be available from the Xtend project for the 2.2.1 release and Eclipse Juno is still some way to go I decided to backport the current 2.3.0 M6 sources of the Xtend Maven Plugin to 2.2.x and create an unofficial version 2.2.2. This plugin I have deployed to the Fornax Repository to make it public available.

For an usage example refer to the POMs of project Spray: pom.xml, parent pom.xml.

Xtext 2.2 finally brings Maven support for Xtend

I cannot tell how often I was asked since introduction of Xtend2 how to compile them in a Maven build. This was just not possible until now due to the problem that to load and compile an Xtend class it is necessary to compile all of the Java classes that the Xtend class on before, and Java classes might depend on Xtend classes to be translated in order to be compilable.

Xtext 2.2.0 was just released yesterday, and for me the most important new feature is direct support through the xtend-maven-plugin plugin. I could not hesitate to test this feature, and created a small example.

Scenario

I have set up a small Maven project with 2 Java classes and 1 Xtend class that depend on each other.

To translate XtendClass1 to Java code and compile it requires that JavaClass1 is compiled before, and to compile JavaClass2 it is necessary that XtendClass is translated to Java code before.

Maven POM Configuration

I have decided to not use Maven Tycho for this example, and use the typical structure of an Eclipse project with usage of Xtend, i.e. sources in folder /src (instead /src/main/java) and xtend-gen to generate the Java code for Xtend classes to.

Repositories

Since shortly Xtext artifacts are available on maven.eclipse.org. The new Maven plugin is available on http://build.eclipse.org/common/xtend/maven/

	<repositories>
		<repository>
			<id>maven.eclipse.org</id>
			<url>http://maven.eclipse.org/nexus/content/groups/public/</url>
		</repository>
		<repository>
			<id>xtend</id>
			<url>http://build.eclipse.org/common/xtend/maven/</url>
		</repository>
	</repositories>
	<pluginRepositories>
		<pluginRepository>
			<id>xtend</id>
			<url>http://build.eclipse.org/common/xtend/maven/</url>
		</pluginRepository>
		<pluginRepository>
			<id>fornax</id>
			<url>http://fornax-platform.org/nexus/content/groups/public/</url>
		</pluginRepository>
	</pluginRepositories>

Dependencies

This small project setup requires just a mimimal set of dependencies to be configured:

	<dependencies>
		<dependency>
			<groupId>org.eclipse.xtend2</groupId>
			<artifactId>org.eclipse.xtend2.lib</artifactId>
			<version>2.2.0</version>
		</dependency>
		<dependency>
			<groupId>org.eclipse.xtext</groupId>
			<artifactId>org.eclipse.xtext.xtend2.lib</artifactId>
			<version>2.2.0.v201112061305</version>
		</dependency>
		<dependency>
			<groupId>com.google.inject</groupId>
			<artifactId>com.google.inject</artifactId>
			<version>2.0.0.v201105231817</version>
		</dependency>
	</dependencies>

Source folders

The main source folder src/ can be configured with the build/sourceDirectory setting, for xtend-gen we need the support of the build-helper-maven-plugin.

	<build>
		<sourceDirectory>src</sourceDirectory>
		<plugins>
			<plugin>
				<groupId>org.codehaus.mojo</groupId>
				<artifactId>build-helper-maven-plugin</artifactId>
				<version>1.7</version>
				<executions>
					<execution>
						<id>add-source</id>
						<phase>generate-sources</phase>
						<goals>
							<goal>add-source</goal>
						</goals>
						<configuration>
							<sources>
								<source>xtend-gen</source>
							</sources>
						</configuration>
					</execution>
				</executions>
			</plugin>
			...

The xtend-gen folder must be emptied when executing mvn clean, this requires some additional configuration of the maven-clean-plugin.

			<plugin>
				<artifactId>maven-clean-plugin</artifactId>
				<version>2.4.1</version>
				<configuration>
					<filesets>
						<fileset>
							<directory>xtend-gen</directory>
							<includes>
								<include>**</include>
							</includes>
						</fileset>
					</filesets>
				</configuration>
			</plugin>

xtend-maven-plugin

Now let’s finally come to the Xtend plugin. The plugin is configured as follows:

			<plugin>
				<groupId>org.eclipse.xtend2</groupId>
				<artifactId>xtend-maven-plugin</artifactId>
				<version>2.2.2</version>
				<executions>
					<execution>
						<goals>
							<goal>compile</goal>
							<goal>testCompile</goal>
						</goals>
						<configuration>
							<outputDirectory>xtend-gen</outputDirectory>
						</configuration>
					</execution>
				</executions>
			</plugin>

The outputDirectory is optional, but leaving it out the sources are generated to src/main/xtend-gen, and we want to generate to xtend-gen in the project root.

The plugin does not define a lifecycle mapping for M2E, which leads to an error marker “Plugin execution not covered by lifecycle configuration”. This is a well known issue when using M2E, and requires some additional work on the plugin. I have opened Bug#366118 for this.

Sample Project

The project described above is shared on Github: https://github.com/kthoms/xtext-experimental

In Eclipse the project looks like this after import (assuming the M2E plugin installed):

Build Execution

The classes in the project can now be built using mvn clean install. This will produce the following output:

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building xtend-maven-classic 2.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ xtend-maven-classic ---
[INFO] Deleting /Users/thoms/git/xtext-experimental/maven/xtend.maven/target
[INFO] Deleting /Users/thoms/git/xtext-experimental/maven/xtend.maven/xtend-gen (includes = [**], excludes = [])
[INFO] 
[INFO] --- build-helper-maven-plugin:1.7:add-source (add-source) @ xtend-maven-classic ---
[INFO] Source directory: /Users/thoms/git/xtext-experimental/maven/xtend.maven/xtend-gen added.
[INFO] 
[INFO] --- xtend-maven-plugin:2.2.0:compile (default) @ xtend-maven-classic ---
[WARNING] 
WARNING: 	XtendClass1.xtend - /Users/thoms/git/xtext-experimental/maven/xtend.maven/src/mypackage/XtendClass1.xtend
7: The value of the field XtendClass1.cls is not used
[INFO] Compiling 1 source file to xtend-gen
[INFO] 
[INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ xtend-maven-classic ---
[WARNING] Using platform encoding (MacRoman actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/thoms/git/xtext-experimental/maven/xtend.maven/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ xtend-maven-classic ---
[WARNING] File encoding has not been set, using platform encoding MacRoman, i.e. build is platform dependent!
[INFO] Compiling 3 source files to /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.923s
[INFO] Finished at: Thu Dec 08 23:01:42 CET 2011
[INFO] Final Memory: 20M/81M
[INFO] ------------------------------------------------------------------------

From the output it can be seen that at the end 3 Java classes are compiled, after the xtend-maven-plugin created the Java code for the Xtend class.

Enabling debug output with -X reveals some more insight, how the plugin works:

[DEBUG] Configuring mojo 'org.eclipse.xtend2:xtend-maven-plugin:2.2.0:compile' with basic configurator -->
[DEBUG]   (f) outputDirectory = xtend-gen
[DEBUG]   (f) project = MavenProject: org.eclipse.xtext.example:xtend-maven-classic:2.2.0-SNAPSHOT @ /Users/thoms/git/xtext-experimental/maven/xtend.maven/pom.xml
[DEBUG]   (f) tempDirectory = /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend
[DEBUG] -- end configuration --
[DEBUG] load xtend file 'file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/mypackage/XtendClass1.xtend'
[DEBUG] Parsing took: 42 ms
...
[DEBUG] create java stub 'mypackage/XtendClass1.java'
[DEBUG] invoke batch compiler with '-cp /Users/thoms/git/xtext-experimental/maven/xtend.maven/src:/Users/thoms/.m2/repository/org/eclipse/xtend2/org.eclipse.xtend2.lib/2.2.0/org.eclipse.xtend2.lib-2.2.0.jar:/Users/thoms/.m2/repository/com/google/guava/guava/10.0.1/guava-10.0.1.jar:/Users/thoms/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xtend2.lib/2.2.0.v201112061305/org.eclipse.xtext.xtend2.lib-2.2.0.v201112061305.jar:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xbase.lib/2.2.0.v201112061305/org.eclipse.xtext.xbase.lib-2.2.0.v201112061305.jar:/Users/thoms/.m2/repository/com/google/inject/com.google.inject/2.0.0.v201105231817/com.google.inject-2.0.0.v201105231817.jar -d /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/classes -1.5 -proceedOnError /Users/thoms/git/xtext-experimental/maven/xtend.maven/src /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/stubs'
[DEBUG] classpath used for Xtend compilation : [file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/, file:/Users/thoms/.m2/repository/org/eclipse/xtend2/org.eclipse.xtend2.lib/2.2.0/org.eclipse.xtend2.lib-2.2.0.jar, file:/Users/thoms/.m2/repository/com/google/guava/guava/10.0.1/guava-10.0.1.jar, file:/Users/thoms/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar, file:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xtend2.lib/2.2.0.v201112061305/org.eclipse.xtext.xtend2.lib-2.2.0.v201112061305.jar, file:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xbase.lib/2.2.0.v201112061305/org.eclipse.xtext.xbase.lib-2.2.0.v201112061305.jar, file:/Users/thoms/.m2/repository/com/google/inject/com.google.inject/2.0.0.v201105231817/com.google.inject-2.0.0.v201105231817.jar, file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/, file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/classes/]

The plugin invokes the Xtend Batch compiler through a spawned JVM. The batch compiler is realized in class org.eclipse.xtext.xtend2.compiler.batch.Xtend2BatchCompiler from the newly added plugin org.eclipse.xtext.xtend2.standalone.

After the build XtendClass1.java is generated as expected to xtend-gen. When opening the target folder it can be seen that the plugin produces a Java stub class to target/stubs for the Xtend class. Finally, everything is compiled and the project is clean :-)

Limitation

The plugin only takes one source folder into account, namely the one configured by build/sourceDirectory. This is a problem when you have multiple source folders, which is quite typical for Xtext projects, namely src and src-gen. This is reported as , which is fixed in the meantime. I have deployed a patched version as unofficial release 2.2.2.

Conclusion

I deeply desired this plugin and this finally allows that the xtend-gen folder does not need to be checked in. The sample used here was just simple. Next would be a real life project were I want to apply the plugin, most likely in Project Spray. This project uses Xtend based code generation heavily and Maven Tycho for the build. The project has not been upgraded to Xtext 2.2.0 (of course, it was just released), and I guess we have to upgrade to Xtext 2.2.0 before we can use the plugin. But if this allows us to finally remove the xtend-gen folders from the repository this alone is worth upgrading.

Nice work, Xtext team!

Xtext 2.1.1 artifacts available on Eclipse Maven repository

Until now it was always a struggle to make Xtext (or Eclipse) artifacts available on a public available repository. It seems that this has now finally found a good end.

Today Achim Demelt pointed me to a Google+ posting from Dennis Huebner (our Xtext build manager) where he announced that Xtext artifacts are now automatically available on the Nexus repository at maven.eclipse.org.

The given POMs have all dependencies from their bundle manifests. This is the recommended “Maven way”, but you may face that too many dependencies are pulled transitive, and you may have to cut them down by explicit exclusions. I haven’t tested this with the new repository, but this is what I experienced when doing a similar approach earlier with Xtext 2.0.1, whose artifacts are deployed on the Fornax Repository. Use the dependency:tree plugin, or better M2E, to identify possible exclusions.

Xtext, Maven and Source Control

Although I more and more use Git for source control, I face Subversion still frequently in daily life. I follow the rule that I don’t want to check in what can be generated in a headless build, where I usually use Maven. Xtext projects have the source folders src-gen and xtend-gen, where generated sources go to. Further, you will usually have 3 projects: The grammar project, the UI project and a test project.

In this article I will show some guidelines that I follow when checking in Xtext projects into source control.

Handling the src-gen folder

The content of src-gen can be produced during a build, so I do not check in the content of it. What needs to be considered is that is no good idea to ignore the folder completely, since it would be missing when fresh checking out the project. This leads to an incomplete build path, and the MWE2 workflow is not executable from Eclipse until the folder was created. This is because the .mwe2 file needs to be on the project’s classpath, which is not the folder src, but the output folder, bin or target/classes. It will be copied automatically with default workspace settings only when the build path is complete. This behavior can be changed in the workspace / project settings in Java / Compiler / Building.

If you rate down build path problems to warning level or just disable build abortion on errors, the .mwe2 file will be copied anyway to the target folder. If using project specific settings you could even check in these settings and share it in the team.

I prefer to check in the src-gen folder, but avoid that any content gets checked in. To do so, add the folder with svn add, but none of its content! Best you do this immediately after the Xtext project was created. After the folder was scheduled for adding, you can set the svn:ignore property to value “*”.

The xtend-gen folder

At the moment it is not possible to compile .xtend files during a Maven build, so the content of xtend-gen must be checked in and handled like handcrafted sources. To cut this story short: Xtend and Java files depend usually on each other. Xtend files cannot be compiled until Java sources from the same project were compiled, and some Java sources will depend on about-to-be-compiled Xtend sources. So the Java source compilation fails, aborting the build.

I hope to solve this problem soon.

Some ignore patterns

On the root of each of the 3 Xtext projects, set the property svn:ignore to this value:

bin
target
.DS_Store
plugin.xml_gen

Configure Maven clean plugin

When running mvn clean on your project you would like to to clean up all derived resources. The target folder will be removed by default, but this is not enough. The content of the src-gen folders must be cleaned up as well. This can be done by configuring the maven-clean-plugin. One thing to consider is that Xtext has 3 projects. It is not correct to configure the clean plugin for all 3 projects due the lifecycle of a Maven build: The projects will be build after each other, executing all lifecycles per project. If you execute a “mvn clean install” this would have the effect that

  1. clean is called on the grammar project
  2. in phase generate-sources the Xtext implementation classes are produced to all 3 projects
  3. the sources for the grammar project are compiled
  4. now the build continues with the next module, e.g. the UI project
  5. the project will be cleaned, removing the previously generated sources
  6. compilation fails, since the generated sources are missing now

For this reason the clean plugin must be configured only in the grammar project, and it should clean up the sources from all 3 projects. Now comes again a detail related to source control: Since we want to have the src-gen checked in, we have to avoid that the SCM related folders and files are removed. Otherwise this would confuse your SCM system. So these files must be excluded from deletion. This leads to the following configuration:

<plugin>
  <artifactId>maven-clean-plugin</artifactId>
  <configuration>
    <filesets>
      <fileset>
        <directory>src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
      <fileset>
        <directory>../${project.artifactId}.ui/src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
      <fileset>
        <directory>../${project.artifactId}.tests/src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
    </filesets>
  </configuration>
</plugin>

Xtext 2.0 / Xpand 1.1 plugins available as Maven artifacts

It’s now one week after release of Eclipse Indigo, and with that Xtext 2.0 and Xpand 1.1. For those who use Maven without Tycho to build projects using these frameworks it is of course necessary to have all related plugins available in some Maven repository.

With the help of the osgi-to-maven2 project I was now able with some tweaks to create the necessary POMs and deploy scripts for the plugins. What osgi-to-maven2 basically does is to analyze the plugin manifests and create POMs for them including dependencies. The project is not maintained at the moment and has a problem especially with the parsing of the plugins org.eclipse.ui.views and org.eclipse.ui.workbench, which have some strange directives in the Export-Package section where I could not find documentation about. The underlying manifest parser from Springsource (com.springsource.util.osgi) vomits when parsing those manifests, and also upgrading to the latest release does not help. Time to open a bug.


Export-Package: org.eclipse.ui;mandatory:="ui.workb
ench",org.eclipse.ui.about,org.eclipse.ui.actions;ui.workbench=split;
mandatory:="ui.workbench",org.eclipse.ui.activities,org.eclipse.ui.ap
plication,org.eclipse.ui.branding,org.eclipse.ui.browser;ui.workbench
=split
;mandatory:="ui.workbench",org.eclipse.ui.commands,org.eclipse.

Also the osgi-to-maven2 tool does not handle absent optional libraries so far.

The Xtext 2.0, Xpand 1.1 and dependent plugins are available at the Fornax Platform’s repository http://fornax-platform.org/nexus/content/groups/public/ under the groupId org.eclipse.plugins.

Based on the deployed plugins it should be possible to setup a plain Maven build for Xtext 2 based projects.

Using the JDT compiler in normal Maven builds

The Maven Compiler Plugin can be customized to use a different compiler than Javac. One of the options listed for the compilerId setting is “eclipse“. To use it you will have to add a dependency to the plexus-compiler-eclipse (current version: 1.8.1).

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-compiler-plugin</artifactId>
	<configuration>
		<compilerId>eclipse</compilerId>
		<source>1.5</source>
		<target>1.5</target>
		<optimize>true</optimize>
	</configuration>
	<dependencies>
		<dependency>
			<groupId>org.codehaus.plexus</groupId>
			<artifactId>plexus-compiler-eclipse</artifactId>
			<version>1.8.1</version>
			<scope>runtime</scope>
		</dependency>
	</dependencies>
</plugin>

A pitfall here is that the compiler compliance level is not automatically set through the target setting. You have to set also optimize=true. Otherwise Java 1.5 sources won’t compile.

The plexus-compiler-eclipse (version 1.8.1) takes a rather old version of JDT, 3.3.0-v_771. To use another version you will have to deploy a newer version of JDT yourself into your repository and list this dependency before plexus-compiler-eclipse.