How to limit proposed Java types to implementors of an interface

When you develop a DSL that derives from Xbase you likely want to refer to Jvm types through a JvmTypeReference.

CustomBehavior :
    'custom' (name = ID | 'ref' realizedBy=JvmTypeReference) (label = STRING);

If you hit CTRL+SPACE at such a position in the editor, by default you will get all visible Java types proposed:

This is not really useful when it only makes sense to refer to specifc types. If you want to limit the proposals to just implementors of a specific interface, you can customize this in the proposal provider by overriding the method completeJvmParameterizedTypeReference_Type(). The following example from Spray shows limiting the types to implementors of Graphiti’s ICustomFeature interface in the context of Spray’s CustomBehavior:

public class SprayProposalProvider extends AbstractSprayProposalProvider {
    @Inject
    private IJvmTypeProvider.Factory      jvmTypeProviderFactory;
    @Inject
    private ITypesProposalProvider        typeProposalProvider;
    
    @Override
    public void completeJvmParameterizedTypeReference_Type(EObject model, Assignment assignment, ContentAssistContext context, ICompletionProposalAcceptor acceptor) {
        if (EcoreUtil2.getContainerOfType(model, CustomBehavior.class) != null) {
            final IJvmTypeProvider jvmTypeProvider = jvmTypeProviderFactory.createTypeProvider(model.eResource().getResourceSet());
            // Graphiti specific
            final JvmType interfaceToImplement = jvmTypeProvider.findTypeByName(ICustomFeature.class.getName());
            typeProposalProvider.createSubTypeProposals(interfaceToImplement, this, context, SprayPackage.Literals.BEHAVIOR__REALIZED_BY, TypeMatchFilters.canInstantiate(), acceptor);
        } else {
            super.completeJvmParameterizedTypeReference_Type(model, assignment, context, acceptor);
        }
    }

Now you will get the desired subtypes as proposals:

Note that this does only limit proposals to certain types. It does not prevent you from actually referring to another type that you don’t get proposed. This needs to be restricted in your scope provider.
 

Xtext 2.2 finally brings Maven support for Xtend

I cannot tell how often I was asked since introduction of Xtend2 how to compile them in a Maven build. This was just not possible until now due to the problem that to load and compile an Xtend class it is necessary to compile all of the Java classes that the Xtend class on before, and Java classes might depend on Xtend classes to be translated in order to be compilable.

Xtext 2.2.0 was just released yesterday, and for me the most important new feature is direct support through the xtend-maven-plugin plugin. I could not hesitate to test this feature, and created a small example.

Scenario

I have set up a small Maven project with 2 Java classes and 1 Xtend class that depend on each other.

To translate XtendClass1 to Java code and compile it requires that JavaClass1 is compiled before, and to compile JavaClass2 it is necessary that XtendClass is translated to Java code before.

Maven POM Configuration

I have decided to not use Maven Tycho for this example, and use the typical structure of an Eclipse project with usage of Xtend, i.e. sources in folder /src (instead /src/main/java) and xtend-gen to generate the Java code for Xtend classes to.

Repositories

Since shortly Xtext artifacts are available on maven.eclipse.org. The new Maven plugin is available on http://build.eclipse.org/common/xtend/maven/

	<repositories>
		<repository>
			<id>maven.eclipse.org</id>
			<url>http://maven.eclipse.org/nexus/content/groups/public/</url>
		</repository>
		<repository>
			<id>xtend</id>
			<url>http://build.eclipse.org/common/xtend/maven/</url>
		</repository>
	</repositories>
	<pluginRepositories>
		<pluginRepository>
			<id>xtend</id>
			<url>http://build.eclipse.org/common/xtend/maven/</url>
		</pluginRepository>
		<pluginRepository>
			<id>fornax</id>
			<url>http://fornax-platform.org/nexus/content/groups/public/</url>
		</pluginRepository>
	</pluginRepositories>

Dependencies

This small project setup requires just a mimimal set of dependencies to be configured:

	<dependencies>
		<dependency>
			<groupId>org.eclipse.xtend2</groupId>
			<artifactId>org.eclipse.xtend2.lib</artifactId>
			<version>2.2.0</version>
		</dependency>
		<dependency>
			<groupId>org.eclipse.xtext</groupId>
			<artifactId>org.eclipse.xtext.xtend2.lib</artifactId>
			<version>2.2.0.v201112061305</version>
		</dependency>
		<dependency>
			<groupId>com.google.inject</groupId>
			<artifactId>com.google.inject</artifactId>
			<version>2.0.0.v201105231817</version>
		</dependency>
	</dependencies>

Source folders

The main source folder src/ can be configured with the build/sourceDirectory setting, for xtend-gen we need the support of the build-helper-maven-plugin.

	<build>
		<sourceDirectory>src</sourceDirectory>
		<plugins>
			<plugin>
				<groupId>org.codehaus.mojo</groupId>
				<artifactId>build-helper-maven-plugin</artifactId>
				<version>1.7</version>
				<executions>
					<execution>
						<id>add-source</id>
						<phase>generate-sources</phase>
						<goals>
							<goal>add-source</goal>
						</goals>
						<configuration>
							<sources>
								<source>xtend-gen</source>
							</sources>
						</configuration>
					</execution>
				</executions>
			</plugin>
			...

The xtend-gen folder must be emptied when executing mvn clean, this requires some additional configuration of the maven-clean-plugin.

			<plugin>
				<artifactId>maven-clean-plugin</artifactId>
				<version>2.4.1</version>
				<configuration>
					<filesets>
						<fileset>
							<directory>xtend-gen</directory>
							<includes>
								<include>**</include>
							</includes>
						</fileset>
					</filesets>
				</configuration>
			</plugin>

xtend-maven-plugin

Now let’s finally come to the Xtend plugin. The plugin is configured as follows:

			<plugin>
				<groupId>org.eclipse.xtend2</groupId>
				<artifactId>xtend-maven-plugin</artifactId>
				<version>2.2.2</version>
				<executions>
					<execution>
						<goals>
							<goal>compile</goal>
							<goal>testCompile</goal>
						</goals>
						<configuration>
							<outputDirectory>xtend-gen</outputDirectory>
						</configuration>
					</execution>
				</executions>
			</plugin>

The outputDirectory is optional, but leaving it out the sources are generated to src/main/xtend-gen, and we want to generate to xtend-gen in the project root.

The plugin does not define a lifecycle mapping for M2E, which leads to an error marker “Plugin execution not covered by lifecycle configuration”. This is a well known issue when using M2E, and requires some additional work on the plugin. I have opened Bug#366118 for this.

Sample Project

The project described above is shared on Github: https://github.com/kthoms/xtext-experimental

In Eclipse the project looks like this after import (assuming the M2E plugin installed):

Build Execution

The classes in the project can now be built using mvn clean install. This will produce the following output:

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building xtend-maven-classic 2.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ xtend-maven-classic ---
[INFO] Deleting /Users/thoms/git/xtext-experimental/maven/xtend.maven/target
[INFO] Deleting /Users/thoms/git/xtext-experimental/maven/xtend.maven/xtend-gen (includes = [**], excludes = [])
[INFO] 
[INFO] --- build-helper-maven-plugin:1.7:add-source (add-source) @ xtend-maven-classic ---
[INFO] Source directory: /Users/thoms/git/xtext-experimental/maven/xtend.maven/xtend-gen added.
[INFO] 
[INFO] --- xtend-maven-plugin:2.2.0:compile (default) @ xtend-maven-classic ---
[WARNING] 
WARNING: 	XtendClass1.xtend - /Users/thoms/git/xtext-experimental/maven/xtend.maven/src/mypackage/XtendClass1.xtend
7: The value of the field XtendClass1.cls is not used
[INFO] Compiling 1 source file to xtend-gen
[INFO] 
[INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ xtend-maven-classic ---
[WARNING] Using platform encoding (MacRoman actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/thoms/git/xtext-experimental/maven/xtend.maven/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ xtend-maven-classic ---
[WARNING] File encoding has not been set, using platform encoding MacRoman, i.e. build is platform dependent!
[INFO] Compiling 3 source files to /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.923s
[INFO] Finished at: Thu Dec 08 23:01:42 CET 2011
[INFO] Final Memory: 20M/81M
[INFO] ------------------------------------------------------------------------

From the output it can be seen that at the end 3 Java classes are compiled, after the xtend-maven-plugin created the Java code for the Xtend class.

Enabling debug output with -X reveals some more insight, how the plugin works:

[DEBUG] Configuring mojo 'org.eclipse.xtend2:xtend-maven-plugin:2.2.0:compile' with basic configurator -->
[DEBUG]   (f) outputDirectory = xtend-gen
[DEBUG]   (f) project = MavenProject: org.eclipse.xtext.example:xtend-maven-classic:2.2.0-SNAPSHOT @ /Users/thoms/git/xtext-experimental/maven/xtend.maven/pom.xml
[DEBUG]   (f) tempDirectory = /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend
[DEBUG] -- end configuration --
[DEBUG] load xtend file 'file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/mypackage/XtendClass1.xtend'
[DEBUG] Parsing took: 42 ms
...
[DEBUG] create java stub 'mypackage/XtendClass1.java'
[DEBUG] invoke batch compiler with '-cp /Users/thoms/git/xtext-experimental/maven/xtend.maven/src:/Users/thoms/.m2/repository/org/eclipse/xtend2/org.eclipse.xtend2.lib/2.2.0/org.eclipse.xtend2.lib-2.2.0.jar:/Users/thoms/.m2/repository/com/google/guava/guava/10.0.1/guava-10.0.1.jar:/Users/thoms/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xtend2.lib/2.2.0.v201112061305/org.eclipse.xtext.xtend2.lib-2.2.0.v201112061305.jar:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xbase.lib/2.2.0.v201112061305/org.eclipse.xtext.xbase.lib-2.2.0.v201112061305.jar:/Users/thoms/.m2/repository/com/google/inject/com.google.inject/2.0.0.v201105231817/com.google.inject-2.0.0.v201105231817.jar -d /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/classes -1.5 -proceedOnError /Users/thoms/git/xtext-experimental/maven/xtend.maven/src /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/stubs'
[DEBUG] classpath used for Xtend compilation : [file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/, file:/Users/thoms/.m2/repository/org/eclipse/xtend2/org.eclipse.xtend2.lib/2.2.0/org.eclipse.xtend2.lib-2.2.0.jar, file:/Users/thoms/.m2/repository/com/google/guava/guava/10.0.1/guava-10.0.1.jar, file:/Users/thoms/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar, file:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xtend2.lib/2.2.0.v201112061305/org.eclipse.xtext.xtend2.lib-2.2.0.v201112061305.jar, file:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xbase.lib/2.2.0.v201112061305/org.eclipse.xtext.xbase.lib-2.2.0.v201112061305.jar, file:/Users/thoms/.m2/repository/com/google/inject/com.google.inject/2.0.0.v201105231817/com.google.inject-2.0.0.v201105231817.jar, file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/, file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/classes/]

The plugin invokes the Xtend Batch compiler through a spawned JVM. The batch compiler is realized in class org.eclipse.xtext.xtend2.compiler.batch.Xtend2BatchCompiler from the newly added plugin org.eclipse.xtext.xtend2.standalone.

After the build XtendClass1.java is generated as expected to xtend-gen. When opening the target folder it can be seen that the plugin produces a Java stub class to target/stubs for the Xtend class. Finally, everything is compiled and the project is clean :-)

Limitation

The plugin only takes one source folder into account, namely the one configured by build/sourceDirectory. This is a problem when you have multiple source folders, which is quite typical for Xtext projects, namely src and src-gen. This is reported as , which is fixed in the meantime. I have deployed a patched version as unofficial release 2.2.2.

Conclusion

I deeply desired this plugin and this finally allows that the xtend-gen folder does not need to be checked in. The sample used here was just simple. Next would be a real life project were I want to apply the plugin, most likely in Project Spray. This project uses Xtend based code generation heavily and Maven Tycho for the build. The project has not been upgraded to Xtext 2.2.0 (of course, it was just released), and I guess we have to upgrade to Xtext 2.2.0 before we can use the plugin. But if this allows us to finally remove the xtend-gen folders from the repository this alone is worth upgrading.

Nice work, Xtext team!

Xtext 2.1.1 artifacts available on Eclipse Maven repository

Until now it was always a struggle to make Xtext (or Eclipse) artifacts available on a public available repository. It seems that this has now finally found a good end.

Today Achim Demelt pointed me to a Google+ posting from Dennis Huebner (our Xtext build manager) where he announced that Xtext artifacts are now automatically available on the Nexus repository at maven.eclipse.org.

The given POMs have all dependencies from their bundle manifests. This is the recommended “Maven way”, but you may face that too many dependencies are pulled transitive, and you may have to cut them down by explicit exclusions. I haven’t tested this with the new repository, but this is what I experienced when doing a similar approach earlier with Xtext 2.0.1, whose artifacts are deployed on the Fornax Repository. Use the dependency:tree plugin, or better M2E, to identify possible exclusions.

EclipseCon 2011 warmup

EclipseCon is just about to start and I am glad that I can make it this year. I have the chance to present Project Spray on thursday (Nov 3rd, 11:30 AM Bürgersaal 2) together with Jos Warmer. Spray aims to provide tooling to facilitate the creation of visual editors based on the Graphiti framework by the use of Xtext based DSL and Xtend code generator. You can basically compare it with what GMF Tools does for GMF, just with a DSL based approach.

EclipseCon Europe 2011

The project was founded at this year’s Code Generation conference, and Jos contributed an initial state of Spray derived from a customer’s project back in August. Since then the project team refactored the code base quite a bit. We are still in an early project phase. Most of the development is done in spare time yet, like Open Source projects often start. That was quite tough besides my full packed work schedule. Many hours on train or evenings in the hotel were spent to push this project.

At the moment I am just about to release version 0.3.0 of the Spray SDK, which I will use for our demo on thursday. Of course we want to gain a large audience. I am honest enough to say that attendees won’t see a production ready tool yet, but of course it is in working state. Jan announced a pie fight for visual tooling, I am ready to open my mouth wide enough to catch the incoming pies.

For our session we don’t want to start any discussion on graphical editing vs. visualization, whether GMF is better than Graphiti, or DSL / code generation vs. framework approach. There is always a niche for everything, and we are confident that Spray fills one of these gaps.

Besides what an end user actually can do with Spray at this moment there are quite some hidden gems in the project. I am using Spray to have also a non-trivial example for Eclipse Modeling tooling. You will find, for example

At EclipseCon I will take the chance to exchange with some experts in the fields of Modeling, especially Xtext and Graphiti. I will lay another focus on Build systems, especially Maven. Besides that there are many other interesting talks, far more than I can attend. And not to forget to meet all the other Eclipse enthusiasts. See you there!

Xtext, Maven and Source Control

Although I more and more use Git for source control, I face Subversion still frequently in daily life. I follow the rule that I don’t want to check in what can be generated in a headless build, where I usually use Maven. Xtext projects have the source folders src-gen and xtend-gen, where generated sources go to. Further, you will usually have 3 projects: The grammar project, the UI project and a test project.

In this article I will show some guidelines that I follow when checking in Xtext projects into source control.

Handling the src-gen folder

The content of src-gen can be produced during a build, so I do not check in the content of it. What needs to be considered is that is no good idea to ignore the folder completely, since it would be missing when fresh checking out the project. This leads to an incomplete build path, and the MWE2 workflow is not executable from Eclipse until the folder was created. This is because the .mwe2 file needs to be on the project’s classpath, which is not the folder src, but the output folder, bin or target/classes. It will be copied automatically with default workspace settings only when the build path is complete. This behavior can be changed in the workspace / project settings in Java / Compiler / Building.

If you rate down build path problems to warning level or just disable build abortion on errors, the .mwe2 file will be copied anyway to the target folder. If using project specific settings you could even check in these settings and share it in the team.

I prefer to check in the src-gen folder, but avoid that any content gets checked in. To do so, add the folder with svn add, but none of its content! Best you do this immediately after the Xtext project was created. After the folder was scheduled for adding, you can set the svn:ignore property to value “*”.

The xtend-gen folder

At the moment it is not possible to compile .xtend files during a Maven build, so the content of xtend-gen must be checked in and handled like handcrafted sources. To cut this story short: Xtend and Java files depend usually on each other. Xtend files cannot be compiled until Java sources from the same project were compiled, and some Java sources will depend on about-to-be-compiled Xtend sources. So the Java source compilation fails, aborting the build.

I hope to solve this problem soon.

Some ignore patterns

On the root of each of the 3 Xtext projects, set the property svn:ignore to this value:

bin
target
.DS_Store
plugin.xml_gen

Configure Maven clean plugin

When running mvn clean on your project you would like to to clean up all derived resources. The target folder will be removed by default, but this is not enough. The content of the src-gen folders must be cleaned up as well. This can be done by configuring the maven-clean-plugin. One thing to consider is that Xtext has 3 projects. It is not correct to configure the clean plugin for all 3 projects due the lifecycle of a Maven build: The projects will be build after each other, executing all lifecycles per project. If you execute a “mvn clean install” this would have the effect that

  1. clean is called on the grammar project
  2. in phase generate-sources the Xtext implementation classes are produced to all 3 projects
  3. the sources for the grammar project are compiled
  4. now the build continues with the next module, e.g. the UI project
  5. the project will be cleaned, removing the previously generated sources
  6. compilation fails, since the generated sources are missing now

For this reason the clean plugin must be configured only in the grammar project, and it should clean up the sources from all 3 projects. Now comes again a detail related to source control: Since we want to have the src-gen checked in, we have to avoid that the SCM related folders and files are removed. Otherwise this would confuse your SCM system. So these files must be excluded from deletion. This leads to the following configuration:

<plugin>
  <artifactId>maven-clean-plugin</artifactId>
  <configuration>
    <filesets>
      <fileset>
        <directory>src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
      <fileset>
        <directory>../${project.artifactId}.ui/src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
      <fileset>
        <directory>../${project.artifactId}.tests/src-gen</directory>
        <includes>
          <include>**</include>
        </includes>
        <excludes>
          <exclude>.svn/**</exclude>
          <exclude>.gitignore</exclude>
        </excludes>
      </fileset>
    </filesets>
  </configuration>
</plugin>

Moving an Xtend generator into its own plugin

One of the nice things that you get when starting an Xtext project is an Xtend based generator that is automatically invoked when you save an Xtext model file. The Xtend generator for your language resides in the .generator subpackage of your language. The problem with that is that it is usually no good idea to have the generator bundled with your language. It is a completely seperate feature which is reasonable to put it in an own plugin. Further, the DSL plugins must not depend on the generator plugin, the dependency must be vice versa. This article describes the steps that need to be done for this.

As a reference take the sources from project Spray. There you can find an concrete example where the described steps have been applied.

Create the generator plugins

Create two additional plugins: One for the runtime part of the generator, one for the UI contributions.

Generator runtime plugin

Move everything from the .generator subpackage of your DSL runtime project to the generator plugin. (In Spray this is org.eclipselabs.spray.generator.graphiti). The dependencies are the same as in the DSL runtime project, but add the DSL runtime plugin as additional dependency (see this MANIFEST.MF as reference). Don’t forget to add an xtend-gen source folder to the project.

Guice module for runtime plugin

Create a Module class extending AbstractGenericModule in the runtime plugin. In this module at least the IGenerator implementation class must be bound.

public class GraphitiGeneratorModule extends AbstractGenericModule {
    public Class<? extends org.eclipse.xtext.generator.IGenerator> bindIGenerator() {
        return SprayGenerator.class;
    }
    ...
}

Generator UI plugin

The UI plugin must basically bind the JavaProjectBasedBuilderParticipant in its own module and register it through the org.eclipse.xtext.builder.participant extension point. These steps are required:

UI Guice Module

Create a module class extending AbstractGenericModule and bind JavaProjectBasedBuilderParticipant:

public class GraphitiGeneratorUIModule extends AbstractGenericModule {
    private final AbstractUIPlugin plugin;
    
    public GraphitiGeneratorUIModule (AbstractUIPlugin plugin) {
        this.plugin = plugin;
    }

    @Override
    public void configure(Binder binder) {
        super.configure(binder);
        binder.bind(AbstractUIPlugin.class).toInstance(plugin);
        binder.bind(IDialogSettings.class).toInstance(plugin.getDialogSettings());
    }
    /**
     * Bind the JavaProjectBasedBuilderParticipant in order to invoke the generator during the build.
     */
    public Class<? extends org.eclipse.xtext.builder.IXtextBuilderParticipant> bindIXtextBuilderParticipant() {
        return org.eclipse.xtext.builder.JavaProjectBasedBuilderParticipant.class;
    }

    ...
}

Activator class

Create a class Activator which creates a Guice injector from the modules:

  1. DSL runtime module
  2. org.eclipse.xtext.ui.shared.SharedStateModule
  3. DSL UI module
  4. Generator runtime module
  5. Generator UI module
public class Activator extends AbstractUIPlugin {
	private Injector injector;
	private static Activator INSTANCE;

	public Injector getInjector() {
		return injector;
	}
	
	@Override
	public void start(BundleContext context) throws Exception {
		super.start(context);
		INSTANCE = this;
		try {
		    injector = Guice.createInjector(Modules2.mixin(new SprayRuntimeModule(), new SharedStateModule(), new SprayUiModule(this), new GraphitiRuntimeModule(), new GraphitiGeneratorModule(), new GraphitiGeneratorUIModule(this)));
		} catch (Exception e) {
			Logger.getLogger(getClass()).error(e.getMessage(), e);
			throw e;
		}
	}
	
	@Override
	public void stop(BundleContext context) throws Exception {
		injector = null;
		super.stop(context);
	}
	
	public static Activator getInstance() {
		return INSTANCE;
	}
	
}

Open the manifest editor. On the “Overview” page enter the Activator class name. Also check the options “Activate this plug-in when one of its classes is loaded” and “This plug-in is a singleton”.

ExecutableExtensionFactory

Create a class ExecutableExtensionFactory. Since this class won’t be public API it is a good approach to put it into an internal package.

public class ExecutableExtensionFactory extends AbstractGuiceAwareExecutableExtensionFactory {

	@Override
	protected Bundle getBundle() {
		return Activator.getInstance().getBundle();
	}
	
	@Override
	protected Injector getInjector() {
		return Activator.getInstance().getInjector();
	}
	
}

plugin.xml

Add a plugin.xml with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.4"?>
<plugin>
   <extension
         point="org.eclipse.xtext.builder.participant">
      <participant
            class="org.eclipselabs.spray.generator.graphiti.ui.internal.ExecutableExtensionFactory:org.eclipse.xtext.builder.IXtextBuilderParticipant">
      </participant>
   </extension>
</plugin>

Check that the ExecutableExtensionFactory class name matches yours.

Remove the GeneratorFragment

Open the .mwe2 workflow of your DSL project. Remove the entry for the GeneratorFragment:

// Code generator
fragment = generator.GeneratorFragment {
  generateJavaMain = false 
  generateMwe = false
  generatorStub = true 
}

Now regenerate the DSL.

Add binding for IWorkspaceRoot

The generator fragment contributes a binding for IWorkspaceRoot to the UI Module of your DSL, which is required by the builder participant. Therefore this binding must be added manually to your UI Module.

public class SprayUiModule extends AbstractSprayUiModule {
    ...
    public org.eclipse.core.resources.IWorkspaceRoot bindIWorkspaceRootToInstance() {
        return org.eclipse.core.resources.ResourcesPlugin.getWorkspace().getRoot();
    }
}

Result

As a result your DSL plugins should have no dependencies on your generator plugin. When you save a model file in your Eclipse instance with the deployed plugins the code generator should be invoked. The pattern described here would also allow to create multiple generator plugins for the same DSL which are invoked independently when building the project. Each of them registers its own builder participant and invokes its own generator.

Setting template file encoding in Xpand workflow

A typical problem when running Xpand is file encoding of the Xpand templates. Xpand uses guillemot brackets («»), which are not available in all character sets. Often, ISO-8859-1 and UTF-8 are good choices for encoding Xpand templates. However, on Mac the default encoding is MacRoman, and I would recommend to change that for interoperability reason. I would recommend changing the resource encoding setting on the root folder containing Xpand templates, or even the project, and checking in the settings. This way the encoding is shared amongst team members.

When executing the Xpand generator with MWE it is important that the templates are read with the right encoding. Otherwise you will get an error like this:
org.eclipse.internal.xtend.xtend.parser.ParseException: no viable alternative at character '�' on line 159
Xpand’s Generator component has a property fileEncoding, but this is the encoding used by the outlets to write files. If you need to change the behavior for reading files, you need to set the encoding value of the ResourceManager. The configuration is like this:

MWE1:


    ...




MWE2

  component = org.eclipse.xpand2.Generator {
    ...
    resourceManager = org.eclipse.xtend.expression.ResourceManagerDefaultImpl {
     	fileEncoding = "ISO-8859-1"
    }
  }

Using Xtext injected types in Xpand

During a project migration I stumbled over the need to access the IScopeProvider from an Xtext based language within Xtend(1) extensions that are used from Xpand templates. This is not quite how Xpand would work out-of-the-box, so here is an approach to solve it.

InjectableGenerator

I subclassed org.eclipse.xpand2.Generator and called it InjectableGenerator. The intention is to register a global variable for a configurable set of types (e.g. the IScopeProvider). The name of the variable is the class name that should be get from Guice. To add injectors, instances of ISetup are registered to the component, similar to Xtext’s org.eclipse.xtext.mwe.Reader component.

The hook that is used to register the global variables to Xpand’s execution context is the method getGlobalVars(WorkflowContext ctx), which is overridden here.

package org.eclipselabs.xtext.xpand;

import java.util.List;
import java.util.Map;
import java.util.Set;

import org.eclipse.emf.mwe.core.WorkflowContext;
import org.eclipse.xpand2.Generator;
import org.eclipse.xtend.expression.Variable;
import org.eclipse.xtext.ISetup;

import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import com.google.inject.ConfigurationException;
import com.google.inject.Injector;

/**
 * Allows Xtext based dependency injection. Injected instances are registered as
 * global vars to the Xpand Execution Context.
 */
public class InjectableGenerator extends Generator {
	private List<Injector> injectors = Lists.newArrayList();

	public void addRegister(ISetup setup) {
		injectors.add(setup.createInjectorAndDoEMFRegistration());
	}

	protected List<Injector> getInjectors() {
		return injectors;
	}
	
	private Set<Class<?>> injectedTypes = Sets.newHashSet();

	public void addInjectGlobalVar (String className) {
		try {
			Class<?> clazz = Class.forName(className);
			injectedTypes.add(clazz);
		} catch (Exception e) {
			throw new IllegalArgumentException("Invalid type: "+className);
		}
	}
	
	@Override
	protected Map<String, Variable> getGlobalVars(WorkflowContext ctx) {
		Map<String, Variable> result = super.getGlobalVars(ctx);
		// try to get an instance of each configured type
		for (Class<?> clazz : injectedTypes) {
			boolean configured = false;
			for (Injector injector : injectors) {
				try {
					Object obj = injector.getInstance(clazz);
					result.put(clazz.getName(), new Variable(clazz.getName(), obj));
					configured = true;
				} catch (ConfigurationException e) {
					; // ignore
				}
			}
			if (!configured) {
				throw new IllegalStateException("Could not configure instance of "+clazz.getName());
			}
		}
		return result;
	}
}

Workflow Configuration

In your workflow configuration you would of course now use the specialized Generator component. It could be used like this:

...
component = org.eclipselabs.xtext.xpand.InjectableGenerator {
   register = my.dsl.MyDslStandaloneSetup {}
   injectGlobalVar = "org.eclipse.xtext.scoping.IScopeProvider"
   // injectGlobalVar = ... other types
   
   // further configuration of Xpand Generator
}

Execution context aware Java extensions

It is not possible to directly access the ExecutionContext from within Xpand templates or Xtend functions. We need to delegate to Java extensions. The classes that we delegate to must implement the IExecutionContextAware interface. The setExecutionContext(ExecutionContext ctx) method is the right place to retrieve the IScopeProvider and bind it to an instance variable.

public class MyExtensions implements IExecutionContextAware {
	protected IScopeProvider scopeProvider;

	@Override
	public void setExecutionContext(ExecutionContext ctx) {
		if (ctx.getGlobalVariables().get("scopeProvider")!=null) {
			this.scopeProvider = (IScopeProvider) ctx.getGlobalVariables().get("scopeProvider").getValue();
		} else if (ctx.getGlobalVariables().get(IScopeProvider.class.getName())!=null) {
			this.scopeProvider = (IScopeProvider) ctx.getGlobalVariables().get(IScopeProvider.class.getName()).getValue();
		} else {
			throw new IllegalStateException("No IScopeProvider found as global var in the execution context");
		}
	}

	// add (non-static!) extension methods here!
        public List<EObject> someMethod (org.eclipse.emf.ecore.EObject context) {
          ... do something
        }
}

Note that we also query the global variable named “scopeProvider” here. This is because in another context (i.e. Check based validation) the same extension would be used with another execution context. The InjectableExecutionContext binds the IScopeProvider to that name.

It might be a good idea to have an abstract base class for these Extension classes.

Now you can use the methods defined in the Java extension class and access the required instances from your language configuration within.

// file: MyExtensions.ext
List[emf::EObject] someMethod (emf::EObject ctx) : JAVA MyExtensions.someMethod(org.eclipse.emf.ecore.EObject);

Xtend – Generating from multiple input models

Xtext’s integration of Xtend based code generators is nice. If you save changed (or add/remove) DSL model files an incremental build will automatically be triggered and even removes derived artifacts if you remove the source from which it is generated. The IGenerator interface takes a Resource, and you can generate 1 or multiple target artifacts from the model elements within that resource.

However, the API does not allow to create a single artifact from multiple input models. But this is a common use case if you do serious code generation. I will show you how you could extend Xtext’s Domainmodel example to generate a simple text file that lists the name of all Entity instances of all .dmodel files that are reachable in the built project.

Adding an extended generator interface

Create an interface IGenerator2 that extends the IGenerator interface:

public interface IGenerator2 extends IGenerator {
	/**
	 * @param input - the input for which to generate resources
	 * @param fsa - file system access to be used to generate files
	 */
	public void doGenerate(ResourceSet input, IFileSystemAccess fsa);
}

The generator class will later implement this interface and generate a single artifact from all Entity instances in all resources.

Extending the JavaProjectBasedBuilderParticipant

The JavaProjectBasedBuilderParticipant is used by default to invoke the IGenerator implementation that was bound to the runtime module. We will extend this class to add additional logic and inject an IGenerator2 instance that will be invoked once per triggered build.

The builder participant is an UI concept, so create the extended class JavaProjectBasedBuilderParticipant2 in the UI plugin:

package org.eclipse.xtext.builder;

import java.util.List;

import org.eclipse.core.runtime.CoreException;
import org.eclipse.core.runtime.IProgressMonitor;
import org.eclipse.emf.ecore.resource.Resource;
import org.eclipse.xtext.builder.JavaProjectBasedBuilderParticipant;
import org.eclipse.xtext.generator.IFileSystemAccess;
import org.eclipse.xtext.generator.IGenerator2;
import org.eclipse.xtext.resource.IContainer;
import org.eclipse.xtext.resource.IResourceDescription;
import org.eclipse.xtext.resource.IResourceDescription.Delta;
import org.eclipse.xtext.resource.IResourceDescriptions;
import org.eclipse.xtext.resource.impl.ResourceDescriptionsProvider;

import com.google.inject.Inject;

public class JavaProjectBasedBuilderParticipant2 extends JavaProjectBasedBuilderParticipant {
  @Inject
  private ResourceDescriptionsProvider resourceDescriptionsProvider;

  @Inject
  private IContainer.Manager containerManager;

  @Inject (optional=true)
  private IGenerator2 generator;

  protected ThreadLocal<Boolean> buildSemaphor = new ThreadLocal<Boolean>();

  @Override
  public void build(IBuildContext context, IProgressMonitor monitor) throws CoreException {
    buildSemaphor.set(false);
    super.build(context, monitor);
  }

  @Override
  protected void handleChangedContents(Delta delta, IBuildContext context, IFileSystemAccess fileSystemAccess) {
    super.handleChangedContents(delta, context, fileSystemAccess);
    if (!buildSemaphor.get() && generator != null) {
      invokeGenerator(delta, context, fileSystemAccess);
    }
  }

  @Override
  protected void handleDeletion(Delta delta, IBuildContext context, IFileSystemAccess fileSystemAccess) {
    super.handleDeletion(delta, context, fileSystemAccess);
    if (!buildSemaphor.get() && generator != null) {
      invokeGenerator(delta, context, fileSystemAccess);
    }
  }

  private void invokeGenerator (Delta delta, IBuildContext context, IFileSystemAccess fileSystemAccess) {
    buildSemaphor.set(true);
    Resource resource = context.getResourceSet().getResource(delta.getUri(), true);
    if (shouldGenerate(resource, context)) {
      IResourceDescriptions index = resourceDescriptionsProvider.createResourceDescriptions();
      IResourceDescription resDesc = index.getResourceDescription(resource.getURI());
      List<IContainer> visibleContainers = containerManager.getVisibleContainers(resDesc, index);
      for (IContainer c : visibleContainers) {
        for (IResourceDescription rd : c.getResourceDescriptions()) {
          context.getResourceSet().getResource(rd.getURI(), true);
        }
      }

      generator.doGenerate(context.getResourceSet(), fileSystemAccess);
    }
  }
}

Note that IGenerator instance will be invoked once per changed resource, so potentially multiple times. We have to make sure that the IGenerator2 instance will be invoked just once. Therefore a thread local semaphor flag will be queried before the generator is called. In this scenario it does not matter whether a change or deletion should be handled. The generator always processes the whole model once.

The most important thing is to populate the BuildContext’s ResourceSet with all relevant model resources. It is important here collect the resources that are in scope of the built project. The right way to do this is to use the language’s IContainer.Manager to collect the resources that are within the visible containers. Each of them is queried for its contained IResourceDescription instances, from which the Resource instances are created.

Extending the generator class

Now open the generator class DomainmodelGenerator and implement IGenerator2. Add an implementation of the method from the extended interface:

...
// makes allContentsIterable available
import static extension org.eclipse.xtext.xtend2.lib.ResourceExtensions.*
...

class DomainmodelGenerator implements IGenerator2 {
  ...
  override void doGenerate(ResourceSet rs, IFileSystemAccess fsa) {
    val Iterable<Entity> entities = rs.resources.map(r|r.allContentsIterable.filter(typeof(Entity))).flatten

    fsa.generateFile("entities.txt", entities.compile)
  }

  def compile (Iterable<Entity> entities) '''
    «FOR e : entities SEPARATOR ", "»«e.name»«ENDFOR»
  '''

  ...
}

Guice configuration

Now bind the extended builder particpant in the UI module:

public class DomainmodelUiModule extends AbstractDomainmodelUiModule {
	@Override
	public Class<? extends IXtextBuilderParticipant> bindIXtextBuilderParticipant() {
		return JavaProjectBasedBuilderParticipant2.class;
	}
}

… and the generator to the Runtime module:

public class DomainmodelRuntimeModule extends AbstractDomainmodelRuntimeModule {
	public Class<? extends IGenerator2> bindIGenerator2 () {
		return DomainmodelGenerator.class;
	}
}

Result & Conclusion

After restarting your runtime workspace the generator will produce an “entities.txt” file to the src-gen folder of a project with .dmodel files, which will simply list the names of all found Entity instances.

This scenario is quite common, so it would be great if the Xtext framework could provide built-in support for that. I have therefore raised a feature request.

Game Development – A whole new area for Eclipse Modeling?

The Eclipse Modeling Project provides the world’s leading set of tools and frameworks that are used for successfully applying model driven software development techniques in various areas. Successful adoption are known in Enterprise Computing, Embedded System Development, Mobile Development etc. But what about Game Development? I have not heard about Game productions that use Eclipse Modeling or Model Driven Software Development in general so far. I cannot know about all projects in the world, but at least it is an indicator that this development technique is at least not wide adopted in the branch of Game Development.

Game Development is highly complex, developed in multidisciplinary teams under high time pressure and quality requirements. And the complexity is even growing, whilst time pressure also. Time-to-market is everything there. If your game comes too late, you are out. If you don’t use the latest technologies, you are lame. How could such projects ever be successful just by coding and hacking? I could imagine that game developers are just too busy with developing their games in a traditional way to think how they could gain speed and quality by applying software engineering techniques like MDSD.

I would not wonder if they associate MDSD with drawing UML diagrams and wasting time clicking and drawing useless stuff. Model Driven Software Development is everything else than useless. It helps raising the level of abstraction, speeding up development and gaining quality. If applied correctly, of course. Of course they think their kind of software development is special and completely different than other disciplines. But let me say, it’s not the case. Every piece of software has generic parts, schematic parts and parts that don’t fit into one of the previous sections. And for the schematic parts, MDSD can always help. Don’t tell me that a multi-million, mission-critical enterprise project is less challenging than game development.

One of the most promising things for game development can be the usage of Domain Specific Languages (DSLs), especially textual ones. With Xtext 2.0 the development of textual DSLs with tight integration of expression languages and code generators has become easier than ever before. If you don’t ever tried Xtext, do it!

On October 4th there will be an interesting workshop at the 10th International Conference on Entertaining Computing (ICEC 2011) in Vancouver, Canada. The “1st Workshop on Game Development and MDSD” will bring experts from both worlds together. The Call for Position Papers is running now, deadline is July 30th. If you are in one of those businesses, submit a propopsal or attend the workshop. I think both “worlds” can really win a lot from working together. This workshop could be a good start.