Normally Xtext projects are built within the Eclipse IDE. But what do you do if the requirement is to have server side builds for for Xtext based projects, including the Grammar project itself, and of course without checking in generated sources? Setting up a working build process for Eclipse plugin projects is often a hell. I often face this requirement, and this is where Maven is a suitable alternative. This article explains how you can build Xtext based projects with Maven.
At first this requires that you have Maven installed on your machine and that the mvn command is on your path. Type on your shell
mvn -version
and it will answer something like
Maven version: 2.0.10
Java version: 1.6.0_07
OS name: "mac os x" version: "10.5.7" arch: "x86_64" Family: "mac"
This is all you basically need.
Xtext projects consist of
- A grammar project
- An UI project
- A generator project
The aim is to run the generator, which of course requires that at least the grammar project is built. We will also build the UI project, but it won’t result into a deployable plugin yet. I plan to investigate on this later. For the moment the projects are just compiled, packaged, and sources generated.
For this article I created projects using the Xtext project wizard, which creates by default the projects
- org.xtext.example.mydsl
- org.xtext.example.mydsl.ui
- org.xtext.example.mydsl.generator
Additionaly a further project was added:
- org.xtext.example.mydsl.parent.
All projects can be downloaded here as Zip. Your project structure will look like this:
Maven Repositories
The most important thing for Maven based builds is that every artifact and plugin required must be available in a Maven Repository.
This is where the story begins, because Eclipse itself has no Maven repository, and thus every artifact (i.e. plugin jar) required within the build must be made available in a Maven Repository. This is at the moment maintained manually. I just deployed all required artifacts to the Maven Repository at openArchitectureWare.org. It contains now the latest 0.7.1 build of TMF Xtext, M2T Xpand and EMFT MWE.
Next repository we need is the one where to get Maven plugins from. Maven itself has just a small kernel and all features are added by plugins. Most of them are quite common and central available (e.g. Compiler Plugin), others are available in project specific repositories. For running Xpand based code generators the Fornax oAW Maven plugin is used, which is available through the Fornax Maven Repository.
In this example we make us of a small plugin (Replacer Plugin) which is available at the JBoss Maven Repository.
Last but not least the central Maven repository will be used to fetch some common artifacts that are not available in both of them. So we need the following repositories:
POM Files
Maven builds are described by Project Object Models (POM). The POMs are stored in pom.xml files for each subproject. These files are basically the build scripts for the projects and this is where all the magic goes in. Mainly it is “just” adding the right POMs to the projects that enable your projects to be built by Maven. This is totally non invasive. For more information about POMs read the Maven POM Reference.
In this article I show just excerpts from the POMs, so make sure to get the sources. Gaps in the POM are marked with “…”.
Parent Project
The org.xtext.example.parent project is the so called “parent project”. The parent’s POM is parent to all POMs from the submodules. The parent pom.xml
- aggregates all subprojects as modules (Multi-module project)
- declares common used Maven plugins
- declares Maven Repositories to use
Module aggregation
We want to build all submodules when running the build on the parent project. This is done by declaring modules by relative paths. Since the projects are organized within an Eclipse workspace all projects including the parent are on the same level.
<modules>
<module>../org.xtext.example.mydsl</module>
<module>../org.xtext.example.mydsl.generator</module>
<module>../org.xtext.example.mydsl.ui</module>
</modules>
Repository configuration
The required repositories are declared within a profile named “RepositoryManagement”, which is activated by default.
<profiles>
<profile>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<id>RepositoryManagement</id>
<repositories>
<repository>
<id>releases.openarchitectureware.org</id>
<name>openArchitectureWare Release Repository</name>
<url>http://www.openarchitectureware.org/m2</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>releases.archiva.fornax-platform.org</id>
<name>Archiva Managed Release Repository</name>
<url>http://www.fornax-platform.org/archiva/repository/releases</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
...
</pluginRepositories>
</profile>
</profiles>
Plugin configuration
Maven plugins are configured in the build section of a POM. Xtext projects require at least Java 1.5, so the Maven compiler plugin is configured so.
<build>
...
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.5</source>
<target>1.5</target>
</configuration>
</plugin>
...
</plugins>
</build>
Grammar Project
The Grammar project org.xtext.example.mydsl contains a workflow that runs the Xtext generator to create the Xtext implementation artifacts. By default the file is GenerateMyDsl.mwe in package org.xtext.example.
When running a build this generator workflow should be executed, generating the Xtext sources. Afterwards the sources should be compiled.
Project parent and basic descriptor tags
The project’s parent is declared by groupId, artifactId and version in the <parent> section. Since groupId and version should always be the same as the parent they are just inherited. This way the must only declared once.
<parent>
<groupId>org.xtext.example.mydsl</groupId>
<artifactId>mydsl-parent</artifactId>
<version>1.0.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>mydsl-grammar</artifactId>
<name>TMF Xtext Example - Grammar</name>
Dependencies
For generating the sources and compilation the Grammar project needs just 2 dependencies:
<dependencies>
<dependency>
<groupId>org.eclipse.xtext</groupId>
<artifactId>xtext-generator</artifactId>
<version>0.7.1</version>
</dependency>
<dependency>
<groupId>de.itemis.xtext</groupId>
<artifactId>antlr</artifactId>
<version>0.7.1</version>
</dependency>
</dependencies>
All other required dependencies are included by Maven’s transitive dependency management.
Alternative Source/Resource paths
Xtext projects don’t follow Maven’s default paths for Java sources and resources. By default Maven will compile Java sources only from the src/main/java folder and will add src/main/resources. For Xtext projects there are two source folders, /src and /src-gen. To compile both an additional plugin from Codehaus, the Build Helper Maven Plugin, is needed.
<build>
...
<sourceDirectory>src</sourceDirectory>
<resources>
<resource>
<directory>src</directory>
</resource>
<resource>
<directory>src-gen</directory>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.3</version>
<executions>
<execution>
<id>add-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>src-gen</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
Workflow execution
To execute the MWE workflow we need the Fornax oAW Maven Plugin. Maven builds have a lifecycle, where predefined lifecycles exists. The oAW Maven plugin must be executed in the generate-sources phase. Since version 3.0.0 the Fornax plugin supports both workflow engines, oAW 4 Workflow Engine and oAW 5 MWE. Current version is 3.0.1.
To enable execution of MWE workflows the configuration section must be configured with <workflowEngine>mwe</workflowEngine>. The <workflowDescriptor> parameter specifies the path to the workflow file.
<build>
...
<plugins>
<plugin>
<groupId>org.fornax.toolsupport</groupId>
<artifactId>fornax-oaw-m2-plugin</artifactId>
<version>3.0.1</version>
<configuration>
<outletSrcDir>src-gen</outletSrcDir>
<outletSrcOnceDir>src</outletSrcOnceDir>
</configuration>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>run-workflow</goal>
</goals>
<configuration>
<workflowDescriptor>org/xtext/example/GenerateMyDsl.mwe</workflowDescriptor>
<workflowEngine>mwe</workflowEngine>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
...
</build>
UI Project
Configuring the POM for the UI project org.xtext.example.mydsl.ui is pretty similar to the Grammar project. In this project even no workflow execution is required, just source paths must be adjusted to /src and /src-gen (see above, Grammar project).
Dependencies
The UI plugin has 2 dependencies:
- The Grammar project
- org.eclipse.xtext:xtext-ui-common
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>mydsl-grammar</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.xtext</groupId>
<artifactId>xtext-ui-common</artifactId>
<version>0.7.1</version>
</dependency>
</dependencies>
Generator Project
The POM for the Generator project again is similar to the Grammar project. This plugin of course executes a workflow during the build, so the Fornax oAW Maven plugin needs to be configured again. Again the source paths must be configured in the build section, and the Build Helper plugin used. See the Grammar project for this configuration.
Dependencies
The Generator project needs also just 2 dependencies:
- The Grammar project
- org.eclipse.xtext:xtext-util
All other dependencies come in transitive.
<dependencies>
<dependency>
<groupId>${pom.parent.groupId}</groupId>
<artifactId>mydsl-grammar</artifactId>
<version>${pom.parent.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.xtext</groupId>
<artifactId>xtext-util</artifactId>
<version>0.7.1</version>
</dependency>
</dependencies>
Workflow execution
The configuration of the oAW Maven plugin is similar to the Grammar project. Of course the path of the workflow file differs. To avoid unneccessary generator execution the plugin can be configured to just run the workflow when specified resources change. This is done through the checkResources section. In this case the generator will only run when the model file src/model/MyModel.dsl changes.
<build>
...
<plugins>
<plugin>
<groupId>org.fornax.toolsupport</groupId>
<artifactId>fornax-oaw-m2-plugin</artifactId>
<version>3.0.1</version>
<configuration>
<checkResources>
<checkResource>
src/model/MyModel.mydsl
</checkResource>
</checkResources>
<outletSrcDir>src-gen</outletSrcDir>
</configuration>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>run-workflow</goal>
</goals>
<configuration>
<workflowDescriptor>workflow/MyDslGenerator.mwe</workflowDescriptor>
<workflowEngine>mwe</workflowEngine>
</configuration>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
Running the build
Now it is time to build the projects with Maven. Note that we only added the described pom.xml files to the projects that the Xtext Wizard created.
Open a shell, go to the org.xtext.example.parent folder and type
mvn clean install
The Maven build starts…
[INFO] Scanning for projects...
[INFO] Reactor build order:
[INFO] TMF Xtext Example - Parent
[INFO] TMF Xtext Example - Grammar
[INFO] TMF Xtext Example - Generator
[INFO] TMF Xtext Example - UI
[INFO] ------------------------------------------------------------------------
[INFO] Building TMF Xtext Example - Parent
[INFO] task-segment: [clean, install]
[INFO] ------------------------------------------------------------------------
These are the first log statements that the build prints out. You can see from here in which order Maven will build the projects. First the parent project will be built, followed by Grammar project, Generator project and UI project.
Downloading artifacts
When you start with a clean Maven installation Maven will download all plugins and artifacts required for the build.
...
Downloading: http://www.openarchitectureware.org/m2/org/eclipse/emf/ecore/xmi/2.5.0/xmi-2.5.0.pom
583b downloaded
Downloading: http://www.openarchitectureware.org/m2/org/eclipse/m2t/xpand/m2t-xpand-xpand/0.7.1/m2t-xpand-xpand-0.7.1.pom
1K downloaded
Downloading: http://www.openarchitectureware.org/m2/org/eclipse/m2t/xpand/m2t-xpand/0.7.1/m2t-xpand-0.7.1.pom
3K downloaded
Downloading: http://www.openarchitectureware.org/m2/org/eclipse/m2t/xpand/m2t-xpand-xtend/0.7.0/m2t-xpand-xtend-0.7.0.pom
1K downloaded
Downloading: http://www.openarchitectureware.org/m2/org/eclipse/m2t/xpand/m2t-xpand/0.7.0/m2t-xpand-0.7.0.pom
3K downloaded
Downloading: http://www.openarchitectureware.org/m2/org/eclipse/emf/mwe/emf-mwe/0.7.0/emf-mwe-0.7.0.pom
24K downloaded
Downloading: http://www.openarchitectureware.org/m2/org/eclipse/emf/mwe/emf-mwe-core/0.7.0/emf-mwe-core-0.7.0.pom
2K downloaded
...
This is already a lot! All of the downloaded artifacts will be stored in your local repository, which is usually located at ~/.m2/repository. Once the artifacts are downloaded you don’t need to access the internet again. You can than go to offline mode by adding the -o option:
mvn -o clean install
You may need to configure a proxy to access the public Maven repositories. Read “Configuring a proxy” to get more information.
Generating the Xtext artifacts
When running the build on the Grammar project the artifacts from the Xtext grammar file are generated. You will see that the Fornax oAW Maven plugin starts execution and the Xtext generator is run.
[INFO] [fornax-oaw-m2:run-workflow {execution: default}]
[INFO] Fornax oAW/MWE Maven2 Plugin V3.0.1
0 [main] INFO eclipse.emf.mwe.core.WorkflowRunner - --------------------------------------------------------------------------------------
174 [main] INFO eclipse.emf.mwe.core.WorkflowRunner - EMF Modeling Workflow Engine 0.7.1, Build v200907170432
174 [main] INFO eclipse.emf.mwe.core.WorkflowRunner - (c) 2005-2009 openarchitectureware.org and contributors
174 [main] INFO eclipse.emf.mwe.core.WorkflowRunner - --------------------------------------------------------------------------------------
175 [main] INFO eclipse.emf.mwe.core.WorkflowRunner - running workflow: org/xtext/example/GenerateMyDsl.mwe
175 [main] INFO eclipse.emf.mwe.core.WorkflowRunner -
882 [main] INFO lipse.emf.mwe.utils.StandaloneSetup - Registering platform uri '/Users/thoms/Development/workspaces/oaw-v5-test'
2794 [main] INFO e.core.container.CompositeComponent - DirectoryCleaner: cleaning directory '../org.xtext.example.mydsl/src-gen'
2795 [main] INFO ipse.emf.mwe.utils.DirectoryCleaner - Cleaning /Users/thoms/Development/workspaces/oaw-v5-test/org.xtext.example.mydsl/../org.xtext.example.mydsl/src-gen
2820 [main] INFO e.core.container.CompositeComponent - DirectoryCleaner: cleaning directory '../org.xtext.example.mydsl.ui/src-gen'
2820 [main] INFO ipse.emf.mwe.utils.DirectoryCleaner - Cleaning /Users/thoms/Development/workspaces/oaw-v5-test/org.xtext.example.mydsl/../org.xtext.example.mydsl.ui/src-gen
2824 [main] INFO e.core.container.CompositeComponent - Generator
2929 [main] INFO ipse.xtext.generator.LanguageConfig - generating infrastructure for org.xtext.example.MyDsl with fragments : ImplicitRuntimeFragment, ImplicitUiFragment, GrammarAccessFragment, EcoreGeneratorFragment, ParseTreeConstructorFragment, ResourceFactoryFragment, AntlrDelegatingFragment, JavaValidatorFragment, JavaScopingFragment, FormatterFragment, LabelProviderFragment, TransformerFragment, OutlineNodeAdapterFactoryFragment, JavaBasedContentAssistFragment, DelegatingGeneratorFragment
Done!
When the build is through you will get the build summary. All generators are run, sources are compiled, jar files are created and installed into your local repository.
...
...
[INFO] Installing /Users/thoms/Development/workspaces/oaw-v5-test/org.xtext.example.mydsl.ui/target/org.xtext.example.mydsl.mydsl-ui_1.0.0-SNAPSHOT.jar to /tmp/maven-rep/org/xtext/example/mydsl/mydsl-ui/1.0.0-SNAPSHOT/mydsl-ui-1.0.0-SNAPSHOT.jar
[INFO]
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] ------------------------------------------------------------------------
[INFO] TMF Xtext Example - Parent ............................ SUCCESS [4.743s]
[INFO] TMF Xtext Example - Grammar ........................... SUCCESS [24.037s]
[INFO] TMF Xtext Example - Generator ......................... SUCCESS [2.600s]
[INFO] TMF Xtext Example - UI ................................ SUCCESS [2.766s]
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 34 seconds
[INFO] Finished at: Mon Jul 20 15:07:14 CEST 2009
[INFO] Final Memory: 67M/160M
[INFO] ------------------------------------------------------------------------
Of course on first execution this will take much longer, since every needed dependency is downloaded. On second execution no download is needed anymore.
My colleague Heiko Behrens and I will have a session at the upcoming W-JAX conference.
