Xtext, Maven and Source Control

Although I more and more use Git for source control, I face Subversion still frequently in daily life. I follow the rule that I don’t want to check in what can be generated in a headless build, where I usually use Maven. Xtext projects have the source folders src-gen and xtend-gen, where generated sources go to. Further, you will usually have 3 projects: The grammar project, the UI project and a test project.

In this article I will show some guidelines that I follow when checking in Xtext projects into source control.

Handling the src-gen folder

The content of src-gen can be produced during a build, so I do not check in the content of it. What needs to be considered is that is no good idea to ignore the folder completely, since it would be missing when fresh checking out the project. This leads to an incomplete build path, and the MWE2 workflow is not executable from Eclipse until the folder was created. This is because the .mwe2 file needs to be on the project’s classpath, which is not the folder src, but the output folder, bin or target/classes. It will be copied automatically with default workspace settings only when the build path is complete. This behavior can be changed in the workspace / project settings in Java / Compiler / Building.

If you rate down build path problems to warning level or just disable build abortion on errors, the .mwe2 file will be copied anyway to the target folder. If using project specific settings you could even check in these settings and share it in the team.

I prefer to check in the src-gen folder, but avoid that any content gets checked in. To do so, add the folder with svn add, but none of its content! Best you do this immediately after the Xtext project was created. After the folder was scheduled for adding, you can set the svn:ignore property to value “*”.

The xtend-gen folder

At the moment it is not possible to compile .xtend files during a Maven build, so the content of xtend-gen must be checked in and handled like handcrafted sources. To cut this story short: Xtend and Java files depend usually on each other. Xtend files cannot be compiled until Java sources from the same project were compiled, and some Java sources will depend on about-to-be-compiled Xtend sources. So the Java source compilation fails, aborting the build.

I hope to solve this problem soon.

Some ignore patterns

On the root of each of the 3 Xtext projects, set the property svn:ignore to this value:

bin
target
.DS_Store
plugin.xml_gen

Configure Maven clean plugin

When running mvn clean on your project you would like to to clean up all derived resources. The target folder will be removed by default, but this is not enough. The content of the src-gen folders must be cleaned up as well. This can be done by configuring the maven-clean-plugin. One thing to consider is that Xtext has 3 projects. It is not correct to configure the clean plugin for all 3 projects due the lifecycle of a Maven build: The projects will be build after each other, executing all lifecycles per project. If you execute a “mvn clean install” this would have the effect that

  1. clean is called on the grammar project
  2. in phase generate-sources the Xtext implementation classes are produced to all 3 projects
  3. the sources for the grammar project are compiled
  4. now the build continues with the next module, e.g. the UI project
  5. the project will be cleaned, removing the previously generated sources
  6. compilation fails, since the generated sources are missing now

For this reason the clean plugin must be configured only in the grammar project, and it should clean up the sources from all 3 projects. Now comes again a detail related to source control: Since we want to have the src-gen checked in, we have to avoid that the SCM related folders and files are removed. Otherwise this would confuse your SCM system. So these files must be excluded from deletion. This leads to the following configuration:

<plugin>
  <artifactId>maven-clean-plugin</artifactId>
  <configuration>
    <filesets>
      <fileset>
        <directory>src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
      <fileset>
        <directory>../${project.artifactId}.ui/src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
      <fileset>
        <directory>../${project.artifactId}.tests/src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
    </filesets>
  </configuration>
</plugin>

Xtext 2.0 / Xpand 1.1 plugins available as Maven artifacts

It’s now one week after release of Eclipse Indigo, and with that Xtext 2.0 and Xpand 1.1. For those who use Maven without Tycho to build projects using these frameworks it is of course necessary to have all related plugins available in some Maven repository.

With the help of the osgi-to-maven2 project I was now able with some tweaks to create the necessary POMs and deploy scripts for the plugins. What osgi-to-maven2 basically does is to analyze the plugin manifests and create POMs for them including dependencies. The project is not maintained at the moment and has a problem especially with the parsing of the plugins org.eclipse.ui.views and org.eclipse.ui.workbench, which have some strange directives in the Export-Package section where I could not find documentation about. The underlying manifest parser from Springsource (com.springsource.util.osgi) vomits when parsing those manifests, and also upgrading to the latest release does not help. Time to open a bug.


Export-Package: org.eclipse.ui;mandatory:="ui.workb
ench",org.eclipse.ui.about,org.eclipse.ui.actions;ui.workbench=split;
mandatory:="ui.workbench",org.eclipse.ui.activities,org.eclipse.ui.ap
plication,org.eclipse.ui.branding,org.eclipse.ui.browser;ui.workbench
=split
;mandatory:="ui.workbench",org.eclipse.ui.commands,org.eclipse.

Also the osgi-to-maven2 tool does not handle absent optional libraries so far.

The Xtext 2.0, Xpand 1.1 and dependent plugins are available at the Fornax Platform’s repository http://fornax-platform.org/nexus/content/groups/public/ under the groupId org.eclipse.plugins.

Based on the deployed plugins it should be possible to setup a plain Maven build for Xtext 2 based projects.

Using the JDT compiler in normal Maven builds

The Maven Compiler Plugin can be customized to use a different compiler than Javac. One of the options listed for the compilerId setting is “eclipse“. To use it you will have to add a dependency to the plexus-compiler-eclipse (current version: 1.8.1).

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-compiler-plugin</artifactId>
	<configuration>
		<compilerId>eclipse</compilerId>
		<source>1.5</source>
		<target>1.5</target>
		<optimize>true</optimize>
	</configuration>
	<dependencies>
		<dependency>
			<groupId>org.codehaus.plexus</groupId>
			<artifactId>plexus-compiler-eclipse</artifactId>
			<version>1.8.1</version>
			<scope>runtime</scope>
		</dependency>
	</dependencies>
</plugin>

A pitfall here is that the compiler compliance level is not automatically set through the target setting. You have to set also optimize=true. Otherwise Java 1.5 sources won’t compile.

The plexus-compiler-eclipse (version 1.8.1) takes a rather old version of JDT, 3.3.0-v_771. To use another version you will have to deploy a newer version of JDT yourself into your repository and list this dependency before plexus-compiler-eclipse.

Tycho build works also for oAW 4.3 projects

Yesterday I showed how Xtext 1.0 projects can be build with Maven Tycho. So why shouldn’t the same setup don’t work for ‘old’ openArchitectureWare Xpand projects? This works like a charm. I took the simple project that the Xpand project wizard creates and added the POM, and it worked! You can download the sample project here. See the POM below, a setup for other oAW Xpand projects should be rather similar.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>p2.osgi.bundle</groupId>
	<artifactId>org.openarchitectureware.example.xpand.tycho</artifactId>
	<version>1.0.0</version>
	<packaging>eclipse-plugin</packaging>
	<name>org.openarchitectureware.example.xpand.tycho</name>

	<properties>
		<version.tycho>0.9.0</version.tycho>
	</properties>

	<build>
		<resources>
			<resource>
				<directory>src</directory>
			</resource>
		</resources>
		<plugins>
			<plugin>
				<groupId>org.sonatype.tycho</groupId>
				<artifactId>tycho-maven-plugin</artifactId>
				<version>${version.tycho}</version>
				<extensions>true</extensions>
			</plugin>
			<plugin>
				<groupId>org.sonatype.tycho</groupId>
				<artifactId>target-platform-configuration</artifactId>
				<version>${version.tycho}</version>
				<configuration>
					<resolver>p2</resolver>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.fornax.toolsupport</groupId>
				<artifactId>fornax-oaw-m2-plugin</artifactId>
				<version>3.1.0-SNAPSHOT</version>
				<configuration>
					<workflowEngine>oaw</workflowEngine>
					<workflowDescriptor>workflow/generator.oaw</workflowDescriptor>
				</configuration>
				<executions>
					<execution>
						<phase>generate-sources</phase>
						<goals>
							<goal>run-workflow</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<artifactId>maven-clean-plugin</artifactId>
				<version>2.4.1</version>
				<configuration>
					<filesets>
						<fileset>
							<directory>src-gen</directory>
							<includes>
								<include>**/**</include>
							</includes>
						</fileset>
					</filesets>
				</configuration>
			</plugin>
		</plugins>
	</build>
	<repositories>

		<repository>
			<id>p2.openarchitectureware</id>
			<url>http://www.openarchitectureware.org/updatesite/milestone/</url>
			<layout>p2</layout>
		</repository>
		<repository>
			<id>p2.eclipse.helios</id>
			<url>http://download.eclipse.org/releases/helios</url>
			<layout>p2</layout>
		</repository>
		<!-- At the moment the Fornax plugin is only available as snapshot -->
		<repository>
			<id>fornax-snapshots</id>
			<url>http://www.fornax-platform.org/archiva/repository/snapshots/</url>
			<releases>
				<enabled>false</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</repository>
	</repositories>
	<pluginRepositories>
		<pluginRepository>
			<id>fornax-snapshots</id>
			<url>http://www.fornax-platform.org/archiva/repository/snapshots/</url>
			<releases>
				<enabled>false</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</pluginRepository>
	</pluginRepositories>
</project>

Building Xtext projects with Maven Tycho

Last year I showed how Xtext 0.7.2 projects can be build with Maven. It requires the installation of all related plugins as Maven artifacts in a Maven2 repository, which is a rather hard and error prone task. I was asked many times since then whether I would install Xtext 1.0 artifacts or even milestone versions into the openArchitectureWare Maven2 repository. Although I can really understand this requirement my hope was that I could avoid this and show a way to enable the build with Maven Tycho. Now the time has come that Tycho is mature enough and finally I had the time to do the necessary extension for the Fornax Maven Workflow Plugin to support the recently added MWE2 workflow engine, which is used from Xtext now as default. Maven Tycho and the Fornax plugin will allow to build Xtext projects in the most natural way possible.

A basic requirement for software builds is that everything can be build just from the sources. Also it is a good practice that no generated sources are checked in into a source repository. Unfortunately this often does not hold for Xtext projects, since it is a quite hard task to execute the Xtext generator in a build process. Yes, it is possible, but until now users often have to solve it again and again. Search the net, most projects seen there will just check in all the sources. The Fornax plugin and Maven will standardize this behavior.

What do you need to enable Xtext DSL builds with Tycho? Not much. Basically a Maven 3 installation (the beta is enough) and access to the internet to access the public repositories. A better idea is to use a repository manager in the intranet (like Nexus) to act as proxy for the public repositories. You will have to add some POMs to your projects. That’s it.

I’ll show now how to enable the basic project that you get from the Xtext project wizard get build with Maven Tycho.

1. Install Maven 3

Download Maven 3 from Apache, unpack it and put the bin folder on your path. Check the installation by typing ‘mvn –version‘ in a shell. You should something like this:

mvn --version
Apache Maven 3.0-beta-1 (r935667; 2010-04-19 19:00:39+0200)
Java version: 1.6.0_20
Java home: /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
Default locale: de_DE, platform encoding: MacRoman
OS name: "mac os x" version: "10.5.8" arch: "x86_64" Family: "mac"

2. Create the Xtext project

Select in Eclipse File / New / Project / Xtext Project and leave all the defaults. The project name should be org.xtext.example.mydsl for this tutorial. The projects will be created and almost empty for now. The aim is to get everything build by just adding the POMs.

3. Adjust plugin manifests

With the default Manifests the build does not execute successfully. The build fails because it has problems to resolve the logging infrastructure. It is sufficient to add Import-Package entries to resolve SLF4J and Logback. Open the plugin Manifest of the org.xtext.example.mydsl project, go to the Dependencies page and add the following Imported Packages:

  • ch.qos.logback.classic
  • ch.qos.logback.core.joran.spi
  • org.slf4j

For each of them open the Properties dialog and set the “optional” flag. Otherwise you could get problems when deploying these plugins. (Thanks to Christian Amman for this tip!)

Open the Manifest of org.xtext.example.mydsl.generator, add the same packages and additionally

  • org.apache.commons.logging

Again, set the imported packages to optional.

4. Add POM files

Each project will get a POM file. A Parent POM will contain the common settings and will act as a reactor POM also. Normally the Parent POM is placed in the directory above the modules, or here the plugin projects. This would be the workspace root. In this example we will place the Parent POM in the DSL grammar project. To make builds easier this POM will have the default name pom.xml. On the other side we have to add also a POM for the grammar project itself. Since it will be in the same directory we have to find another name, and we will call it pom-grammar.xml. For the other projects (.generator and .ui) we will stay with pom.xml.

4.1 org.xtext.example.mydsl/pom.xml

The Parent POM configures the Tycho plugins and aggregates the grammar project, UI project and the generator. Also the required repositories are configured here. Note that the folder “src” is added as resource path, this will add this folder to the classpath. The workflow execution would fail if the workflow module cannot be found on the classpath.

<!--?xml version="1.0" encoding="UTF-8"?-->
	4.0.0
	p2.osgi.bundle
	org.xtext.example.mydsl.parent
	1.0.0
	pom
	org.xtext.example.mydsl.MyDsl - Parent

		0.9.0

		./pom-grammar.xml
		../org.xtext.example.mydsl.ui
		../org.xtext.example.mydsl.generator

		<!-- The src directory must be named as resource dir to put it on the build classpath. This is required to resolve the 			workflow module named in the .mwe2 file -->

			src

				org.sonatype.tycho
				tycho-maven-plugin
				${version.tycho}
				true

				org.sonatype.tycho
				target-platform-configuration
				${version.tycho}

					p2

					org.fornax.toolsupport
					fornax-oaw-m2-plugin
					3.1.0-SNAPSHOT

						mwe2

							generate-sources
								run-workflow

			p2.eclipse.helios
			http://download.eclipse.org/releases/helios
			p2

		<!-- At the moment the Fornax plugin is only available as snapshot -->

			fornax-snapshots
			http://www.fornax-platform.org/archiva/repository/snapshots/

				false

				true

org.xtext.example.mydsl/pom-grammar.xml

The build of the grammar project needs to execute the workflow src/org/xtext/example/mydsl/GenerateMyDsl.mwe2. All other settings are already managed by the parent.

<!--?xml version="1.0" encoding="UTF-8"?-->
	4.0.0

		p2.osgi.bundle
		org.xtext.example.mydsl.parent
		1.0.0
		../org.xtext.example.mydsl/pom.xml

	org.xtext.example.mydsl
	eclipse-plugin
	org.xtext.example.mydsl.MyDsl - Grammar

				org.fornax.toolsupport
				fornax-oaw-m2-plugin

					src/org/xtext/example/mydsl/GenerateMyDsl.mwe2

org.xtext.example.mydsl.ui/pom.xml

The build configuration of the UI plugin is even easier. Only the reference to the parent and the minimum project coordinates need to be configured.

<!--?xml version="1.0" encoding="UTF-8"?-->
	4.0.0

		p2.osgi.bundle
		org.xtext.example.mydsl.parent
		1.0.0
		../org.xtext.example.mydsl/pom.xml

	org.xtext.example.mydsl.ui
	eclipse-plugin
	org.xtext.example.mydsl.MyDsl - UI

org.xtext.example.mydsl.generator/pom.xml

The generator plugin must execute its workflow. Note that the src directory and the compile directory of the grammar project are set as resource directory.

<!--?xml version="1.0" encoding="UTF-8"?-->
	4.0.0

		p2.osgi.bundle
		org.xtext.example.mydsl.parent
		1.0.0
		../org.xtext.example.mydsl/pom.xml

	org.xtext.example.mydsl.generator
	eclipse-plugin
	org.xtext.example.mydsl.MyDsl - Generator

			src
			<!-- Add the grammar classes to the classpath -->
			../org.xtext.example.mydsl/target/classes

				org.fornax.toolsupport
				fornax-oaw-m2-plugin

					mwe2
					src/workflow/MyDslGenerator.mwe2

5. Execute the build

Now the build should be executable. Run ‘mvn clean install‘ in the root of the org.xtext.example.mydsl project. Here is an excerpt of the console log you should get. The complete log can be viewed here.

mvn install
[INFO] Scanning for projects...
[WARNING] No explicit target runtime environment configuration. Build is platform dependent.
[INFO] Resolving target platform for project MavenProject: p2.osgi.bundle:org.xtext.example.mydsl:1.0.0 @ /Users/thoms/temp/tycho/trunk/trunk/org.xtext.example.mydsl/pom-grammar.xml
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
[INFO] Adding repository http://download.eclipse.org/releases/helios
[INFO] Adding repository http://download.eclipse.org/releases/helios
[WARNING] No explicit target runtime environment configuration. Build is platform dependent.
[INFO] Resolving target platform for project MavenProject: p2.osgi.bundle:org.xtext.example.mydsl.ui:1.0.0 @ /Users/thoms/temp/tycho/trunk/trunk/org.xtext.example.mydsl.ui/pom.xml
[INFO] Adding repository (cached) http://download.eclipse.org/releases/helios
[WARNING] No explicit target runtime environment configuration. Build is platform dependent.
[INFO] Resolving target platform for project MavenProject: p2.osgi.bundle:org.xtext.example.mydsl.generator:1.0.0 @ /Users/thoms/temp/tycho/trunk/trunk/org.xtext.example.mydsl.generator/pom.xml
[INFO] Adding repository (cached) http://download.eclipse.org/releases/helios
[WARNING] No explicit target runtime environment configuration. Build is platform dependent.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] org.xtext.example.mydsl.MyDsl - Parent
[INFO] org.xtext.example.mydsl.MyDsl - Grammar
[INFO] org.xtext.example.mydsl.MyDsl - UI
[INFO] org.xtext.example.mydsl.MyDsl - Generator
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building org.xtext.example.mydsl.MyDsl - Parent 1.0.0
[INFO] ------------------------------------------------------------------------
....
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] org.xtext.example.mydsl.MyDsl - Parent ............ SUCCESS [0.681s]
[INFO] org.xtext.example.mydsl.MyDsl - Grammar ........... SUCCESS [26.418s]
[INFO] org.xtext.example.mydsl.MyDsl - UI ................ SUCCESS [3.103s]
[INFO] org.xtext.example.mydsl.MyDsl - Generator ......... SUCCESS [10.844s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

5. Miscalleneous

5.1 ANTLR 3 download

Xtext relies on ANTLR 3, which cannot be hosted at Eclipse.org due to incompatible licenses. Therefore Xtext has introduced an automatic download (FYI: This behavior is implemented in AntlrToolFacade) that you have to confirm once:

*ATTENTION*
It is recommended to use the ANTLR 3 parser generator (BSD licence - http://www.antlr.org/license.html).
Do you agree to download it (size 1MB) from 'http://download.itemis.com/antlr-generator-3.0.1.jar'? (type 'y' or 'n' and hit enter)

Now an automatic build is not so responsive to questions like these. The plugin will confirm this download automatically.

5.2 Workflow execution problems

If the Maven plugin does not execute the MWE2 workflow properly it is not so responsive at the moment to name the root cause. But when running the Maven command with “-X” it will output the full Java command including classpath to the console. Copy the full command and try to execute it from the same directory. This might help to detect the problem you have.

/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java -classpath /Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xpand/1.0.0.v201006150611/org.eclipse.xpand-1.0.0.v201006150611.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xtend/1.0.0.v201006150611/org.eclipse.xtend-1.0.0.v201006150611.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.mwe.core/1.0.0.v201006150535/org.eclipse.emf.mwe.core-1.0.0.v201006150535.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.ecore/2.6.0.v20100614-1136/org.eclipse.emf.ecore-2.6.0.v20100614-1136.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.common/2.6.0.v20100614-1136/org.eclipse.emf.common-2.6.0.v20100614-1136.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.apache.commons.cli/1.0.0.v20080604-1500/org.apache.commons.cli-1.0.0.v20080604-1500.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.mwe2.runtime/1.0.0.v201006150446/org.eclipse.emf.mwe2.runtime-1.0.0.v201006150446.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/com.ibm.icu/4.2.1.v20100412/com.ibm.icu-4.2.1.v20100412.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xtext/1.0.0.v201006170321/org.eclipse.xtext-1.0.0.v201006170321.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.ecore.xmi/2.5.0.v20100521-1846/org.eclipse.emf.ecore.xmi-2.5.0.v20100521-1846.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xtext.util/1.0.0.v201006170321/org.eclipse.xtext.util-1.0.0.v201006170321.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/com.google.collect/0.8.0.v201006170321/com.google.collect-0.8.0.v201006170321.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/com.google.inject/2.0.0.v201003051000/com.google.inject-2.0.0.v201003051000.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.antlr.runtime/3.0.0.v200803061811/org.antlr.runtime-3.0.0.v200803061811.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.mwe.utils/1.0.0.v201006150535/org.eclipse.emf.mwe.utils-1.0.0.v201006150535.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.mwe2.launch/1.0.0.v201006150907/org.eclipse.emf.mwe2.launch-1.0.0.v201006150907.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.emf.mwe2.language/1.0.0.v201006150907/org.eclipse.emf.mwe2.language-1.0.0.v201006150907.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xtext.common.types/1.0.0.v201006170321/org.eclipse.xtext.common.types-1.0.0.v201006170321.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.eclipse.xtend.typesystem.emf/1.0.0.v201006150611/org.eclipse.xtend.typesystem.emf-1.0.0.v201006150611.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/ch.qos.logback.classic/0.9.19.v20100519-1505/ch.qos.logback.classic-0.9.19.v20100519-1505.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/ch.qos.logback.core/0.9.19.v20100419-1216/ch.qos.logback.core-0.9.19.v20100419-1216.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.apache.commons.logging/1.1.1.v201005080502/org.apache.commons.logging-1.1.1.v201005080502.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.slf4j.api/1.5.11.v20100519-1910/org.slf4j.api-1.5.11.v20100519-1910.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/ch.qos.logback.slf4j/0.9.19.v20100519-1910/ch.qos.logback.slf4j-0.9.19.v20100519-1910.jar:/Users/thoms/.m2/repository/p2/osgi/bundle/org.slf4j.log4j/1.5.11.v20100419-1106/org.slf4j.log4j-1.5.11.v20100419-1106.jar org.eclipse.emf.mwe2.launch.runtime.Mwe2Launcher /Users/thoms/Development/workspaces/oaw-v5-test/org.xtext.example.mydsl.generator/src/workflow/MyDslGenerator.mwe2

5.3 Fornax Plugin Snapshot version

The Maven plugin org.fornax.toolsupport:fornax-oaw-m2-plugin is not released yet. Yesterday I have added the necessary support for MWE2 and deployed a snapshot version 3.1.0-SNAPSHOT. Some minor issues have to be solved before it will be released.

5.4 Missing log output

The plugin forks a JVM with help of the Java ant task type. At the moment the output from the forked JVM does not get redirected to the Plugin’s log. This is one issue to solve for the Plugin before release. It makes debugging problems a bit harder, but when copying the executed Java command that you get from the debug output and execute yourself you get a better impression of the error.

5.5 Download sources

For your convenience you can download the example project here.

Tycho slides uploaded

Yesterday I gave a presentation about Maven 3 and Tycho for the Dortmunder Vortragsreihe, a public event series from itemis. A similar talk I gave at the JAX 2010 conference last week, but with less presentation time and thus a less content. I worked on learning about Tycho now quite a while and some bugs and the current lack of documentation led to some restless evenings in the hotel. At the end I wanted to have a continuous build for a non trivial RCP application build. As the example project I wanted to build I took Kai Tödter’s MP3Manager RCP sample project. Finally I got the project building with Tycho. There are still some issues, especially with the product export. I will use this project now to create situations that show bugs and report them to the Tycho team. They are working really hard at the moment to improve Tycho, and I’m sure that many issues will be resolved soon.

Anyway, the slides that I used yesterday were just uploaded, and I hope they will be useful to get started with Tycho.

Maven 3 / Tycho

In the future I will report more about Tycho, so stay tuned!

Mavenizing the project structure of Xtext projects

The structure of typical Xtext projects does not match the standard layout of Maven projects. Xtext projects more adhere to standard Eclipse project layout, which means

  • manual implemented Java sources are in folder /src
  • the plugin Manifest is /META-INF/MANIFEST.MF
  • generated sources go to /src-gen

In my customer’s project Xtext sources are built with Maven, and also the sources produced by Xtext are produced within the Maven build, using the Fornax Workflow Maven plugin. Until now we have adjusted the Maven build to match the standard Xtext project structure, which requires some configuration in the build section of the POM like follows:

   <build>
        <sourceDirectory>src</sourceDirectory>
        <resources>
            <resource>
                <directory>src</directory>
            </resource>
            <resource>
                <directory>src-gen</directory>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <groupId>org.fornax.toolsupport</groupId>
                <artifactId>fornax-oaw-m2-plugin</artifactId>
                <version>3.0.3</version>
                <configuration>
                    <checkFilesets>
                        <checkFileset>
                            <directory>${basedir}</directory>
                            <includes>
                                <include>src/**</include>
                            </includes>
                            <excludes>
                                <exclude>**/.svn/**</exclude>
                            </excludes>
                        </checkFileset>
                    </checkFilesets>
                    <outletSrcDir>src-gen</outletSrcDir>
                    <outletSrcOnceDir>src</outletSrcOnceDir>
                    <outletResOnceDir>resources</outletResOnceDir>
                    <properties></properties>
                    <workflowDescriptor>xtext/example/GenerateMyDsl.mwe</workflowDescriptor>
                    <workflowEngine>mwe</workflowEngine>
                </configuration>
                <executions>
                    <execution>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>run-workflow</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>build-helper-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>add-source</id>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>add-source</goal>
                        </goals>
                        <configuration>
                            <sources>
                                <source>src</source>
                            </sources>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

Now requirements changed and the project structure should conform to the Maven structure, which means:

  • src/main/java: Contains hand-written Java code
  • src/main/resources: All non-Java hand-written code (including Workflow and Xtext grammar)
  • target/generated/java: Contains generated code

Another requirement  from the build team is that the actual output directory ‘target’ should be configurable. This means mainly that we have to use properties that Maven uses to refer to the source and target directories (project.build.sourceDirectory and project.build.directory), so that the build could override just these settings by passing a system property and all output gets produced to and compiled from an alternative directory structure.

Of course this is possible with small changes, but you have to know where.

Xtext Generator Worklow

In order to produce the output of the Xtext generator to different directories than default (src, src-gen => src/main/java, target/generated/java) the MWE workflow file has to be changed. For the generator component we have to override the properties srcPath and srcGenPath. Further, these output directories should be parametrized by the same properties that Maven uses, namely project.build.directory and project.build.directory. These properties need to be configured in the Generate<MyDSL>.properties with defaults.

Generate<MyDSL>.mwe

<workflow>
  ..
  <component class="org.eclipse.xtext.generator.Generator">
    <pathRtProject value="${runtimeProject}"/>
    <pathUiProject value="${runtimeProject}.ui"/>
    <projectNameRt value="${projectName}"/>
    <projectNameUi value="${projectName}.ui"/>
    <srcPath value="/${project.build.sourceDirectory}"/>
    <srcGenPath value="/${project.build.directory}/generated/java"/>
    ..
  </component>
  ..
</workflow>
<pre>

Generate<MyDSL>.properties

basedir=.
project.build.directory=target
project.build.sourceDirectory=src/main/java
grammarURI=classpath:/.../MyDSL.xtext
file.extensions=...
projectName=...

pom.xml: build section

The Java source directories also contain non-Java resources, like the workflow, grammar file etc. Normally this would go to the resources directory for Maven projects, but Xtext (0.7.2) cannot be configured to produce resource files to another directory than Java sources. On the other side those resources need to be found on the classpath during the build. This requires that we add the Java directories as resource directories in the build section of the POM. This settings already existed before, it just has been adopted.

Before:

    <build>
        <sourceDirectory>src</sourceDirectory>
        <resources>
            <resource>
                <directory>src</directory>
            </resource>
            <resource>
                <directory>src-gen</directory>
            </resource>
        </resources>
        ...
   </build>

After:

    <build>
        <resources>
            <resource>
                <directory>${project.build.sourceDirectory}</directory>
            </resource>
            <resource>
                <directory>${project.build.directory}/generated/java</directory>
            </resource>
        </resources>
         ...
     </build>

The <sourceDirectory> entry is not necessary anymore, since the main Java source directory is now src/main/java, which follows the Maven standard directory layout and gets automatically compiled.

maven-clean-plugin

Xtext produces output into two projects – the grammar project and the UI project. Now if the generated UI sources are produced below /target calling ‘mvn clean install’ would have an undesired side effect. Maven builds both modules after each other, so when the UI module is built with the goals ‘clean install’ the target directory is removed and the previous generated sources get lost.

The solution is that the maven-clean-plugin must be deactivated for the UI module and the grammar module must clean the target directory for the UI module.

pom.xml – Grammar project

<plugin>
    <artifactId>maven-clean-plugin</artifactId>
    <configuration>
        <filesets>
            <fileset>
                <directory>../my.dsl.ui</directory>
                <includes>
                    <include>${project.build.directory}</include>
                </includes>
            </fileset>
        </filesets>
    </configuration>
</plugin>

pom.xml – UI project

<plugin>
    <artifactId>maven-clean-plugin</artifactId>
    <configuration>
        <skip>true</skip>
    </configuration>
</plugin>

Eclipse settings

The new project structure affects, of course, the project settings for Eclipse. The settings apply for the grammar and UI plugin.

build.properties

Change the source folders in the build.properties file:

source.. = src/main/java/,target/generated/java/
bin.includes = META-INF/,\
 .,\
 plugin.xml

Java build path

Open the project properties and change the  Java Build Path settings:

  • Source Folder src -> src/main/java
  • Source Folder src-gen -> target/generated/java
  • Default outputr folder: bin -> target/classes