Xtext Content Assist: Filtering keyword proposals

Sometimes when I introduce template proposals in Xtext editors I want them to be a replacement for a keyword proposal. By default you would get a proposal for the template AND for the keyword. In the following screenshot a template for a “class” concept exists, and a proposal for the “class” keyword, which would be inserted as part of the template at the cursor position.

The key to the solution is to overwrite the completeKeyword() method from your proposal provider class, e.g. like this:

public class SprayProposalProvider extends AbstractSprayProposalProvider {
    private static final Set<String>      FILTERED_KEYWORDS = Sets.newHashSet("text", "line", "class", "behavior", "style", "custom");
    @Override
    public void completeKeyword(Keyword keyword, ContentAssistContext contentAssistContext, ICompletionProposalAcceptor acceptor) {
        if (FILTERED_KEYWORDS.contains(keyword.getValue())) {
            // don't propose keyword
            return;
        }
        super.completeKeyword(keyword, contentAssistContext, acceptor);
    }
}

Christian Dietrich added in a comment how to filter a keyword that may occur more than once in your grammar. If you want to filter specific occurances you need to use the IGrammarAccess of your language:

@Inject MyDslGrammarAccess grammarAccess;

@Override
public void completeKeyword(Keyword keyword,ContentAssistContext contentAssistContext,ICompletionProposalAcceptor acceptor) {
  if (!grammarAccess.getGreetingAccess().getHelloKeyword_0().equals(keyword)) {
    super.completeKeyword(keyword, contentAssistContext, acceptor);
  }
}

Fornax Workflow Plugin 3.4.0 released

Finally I have managed to release a new version of the Fornax Maven Workflow Plugin again. This Maven plugin allows execution of MWE and MWE2 workflows from Maven, which is typically the case if you want to automate the build of Xtext or Xpand based projects with Maven.

The new version has some interesting new features which may be worth upgrading:

• m2e lifecycle metadata
• workflow dependencies can be configured as plugin dependencies
• more options to detect changes that trigger the build
• force/skip parameter to configure workflow execution overrides per configuration
• log level detection when running in forked JVM

I have especially to thank Dieter Schulten from escalon, who has been willing to contribute to this release. He has mainly contributed the m2e configuration and some tests for the plugin.

xtext-utils unittesting 0.9.4 released

I have just released the new version 0.9.4 of the xtext-utils unittesting framework. The release became necessary due to a too restrictive version constraint for Google Guice, which made the framework incompatible with the upcoming Xtext 2.3.0 release.

This little framework is a useful addition to Xtext which makes writing DSL tests quite easy. In my daily work on developing Xtext projects and coaching Xtext in workshops it became a useful tool which complements Xtext’s test support. It is not the only way to write tests for Xtext DSLs, but a real simple one. The main idea is that the best way to develop DSLs is to write prototype models. This framework allows you to read, parse, validate, format, serialize, and compare your models in a single line.

The framework can be installed from its update site.

Unofficial release 2.2.2 of the Xtend Maven Plugin

In my previous post I showed how to use the maven-xtend-plugin to compile Xtend sources within a Maven build. This worked fine in simple cases, but there was a limitation of only one supported source directory (the one configured by build/sourceDirectory property), see Bug#367914. Unfortunately already each Xtext project has already 2 source folders (src and src-gen), so this made the plugin unusable for Xtext projects and forced checking in the generated Java files. This causes terrible problems with version control, especially when working on a team.

In the meantime Bug#367914 was resolved, but only for the upcoming Xtend 2.3 (Eclipse Juno) version. This urgent bugfix was not available on a Maven repository. But I needed it for project Spray to finally get rid of the merge conflicts and delete the generated sources from the repository (issue#94). Spray is still using Xtext/Xtend version 2.2.1 and depends on public available Maven artifacts. Since the bugfix won’t be available from the Xtend project for the 2.2.1 release and Eclipse Juno is still some way to go I decided to backport the current 2.3.0 M6 sources of the Xtend Maven Plugin to 2.2.x and create an unofficial version 2.2.2. This plugin I have deployed to the Fornax Repository to make it public available.

For an usage example refer to the POMs of project Spray: pom.xml, parent pom.xml.

JvmType import check

Several Xtext based DSLs will use import statements that import JvmTypes, be them real Java types, or just inferred types produced with your IJvmModelInferrer. The check for a valid import might be simple, but here it is to just make it even easier for you. I just had the need to realize this to resolve Spray issue#113. Basically you need to get the IJvmTypeProvider, to which you gain access through the IJvmTypeProvider.Factory.

public class SprayJavaValidator extends AbstractSprayJavaValidator implements IssueCodes {
    @Inject
    private IJvmTypeProvider.Factory typeProviderFactory;
    @Check
    public void checkImports(final Import imp) {
        // don't check wildcard imports
        if (imp.getImportedNamespace().endsWith(".*"))
            return;
        IJvmTypeProvider typeProvider = typeProviderFactory.findOrCreateTypeProvider(imp.eResource().getResourceSet());
        JvmType jvmType = typeProvider.findTypeByName(imp.getImportedNamespace());
        if (jvmType == null) {
            error("The import " + imp.getImportedNamespace() + " cannot be resolved", SprayPackage.Literals.IMPORT__IMPORTED_NAMESPACE, IMPORT_NOTEXISTS, new String[0]);
        }
    }
    ...
}

That’s all.

Xtext Content Assist: Escape identifiers conflicting with keywords

I have faced in Spray the situation that a segment of a package name of a qualified Java type conflicts with a keyword of the Spray language. Specifically you can refer to custom features with the custom keyword, and the referred type was in a package containing also the name custom.

When inserting the proposed qualified name this results in an error due to the keyword collision.

The qualified name is usually combined through a sequence of ID rules, which allows escaping an identifier by prefixing it with the ^ character in the case of conflict with a keyword.

For content assist it would now be useful if the inserted string would be automatically escaped. So here is a way to solve this.

My first approach was to override the completeJvmParameterizedTypeReference_Type() method from the proposal provider and modify the proposals through a custom ICompletionProposalAcceptor. Sebastian Zarnekow commented that the proper way would be to register a IValueConverter for QualifiedName. I tried to do so, but was not successful. Actually there is already an QualifiedNameValueConverter registered.

My next solution is now to subclass JdtTypesProposalProvider and pass a custom IValueConverter. The overridden methods just pass null as converter, thus do no conversion. I am reusing the QualifiedNameValueConverter here, since it does the necessary keyword escaping.

public class SprayJdtTypesProposalProvider extends JdtTypesProposalProvider {
    @Inject
    private QualifiedNameValueConverter qnValueConverter;

    /**
     * Overridden to pass a default value converter
     */
    @Override
    public void createSubTypeProposals(JvmType superType, ICompletionProposalFactory proposalFactory, ContentAssistContext context, EReference typeReference, Filter filter, ICompletionProposalAcceptor acceptor) {
        createSubTypeProposals(superType, proposalFactory, context, typeReference, filter, getConverter(), acceptor);
    }

    /**
     * Overridden to pass a default value converter
     */
    @Override
    public void createTypeProposals(ICompletionProposalFactory proposalFactory, ContentAssistContext context, EReference typeReference, Filter filter, ICompletionProposalAcceptor acceptor) {
        createTypeProposals(proposalFactory, context, typeReference, filter, getConverter(), acceptor);
    }

    private AbstractValueConverter<String> getConverter() {
        return new AbstractValueConverter<String>() {
            /**
             * Remove ^ character from escaped segments
             */
            @Override
            public String toValue(String string, INode node) throws ValueConverterException {
                return string.replace("^", "");
            }

            /**
             * Escape segments colliding with keywords with ^ character
             */
            @Override
            public String toString(String value) throws ValueConverterException {
                // this converter will escape keywords with ^
                return qnValueConverter.toString(value);
            }
        };
    }
}

This custom implementation must be bound to the UI module:

public class SprayUiModule extends AbstractSprayUiModule {
    @Override
    public Class<? extends ITypesProposalProvider> bindITypesProposalProvider() {
        return SprayJdtTypesProposalProvider.class;
    }
}

Now the inserted qualified type name will be automatically replaced.

How to limit proposed Java types to implementors of an interface

When you develop a DSL that derives from Xbase you likely want to refer to Jvm types through a JvmTypeReference.

CustomBehavior :
    'custom' (name = ID | 'ref' realizedBy=JvmTypeReference) (label = STRING);

If you hit CTRL+SPACE at such a position in the editor, by default you will get all visible Java types proposed:

This is not really useful when it only makes sense to refer to specifc types. If you want to limit the proposals to just implementors of a specific interface, you can customize this in the proposal provider by overriding the method completeJvmParameterizedTypeReference_Type(). The following example from Spray shows limiting the types to implementors of Graphiti’s ICustomFeature interface in the context of Spray’s CustomBehavior:

public class SprayProposalProvider extends AbstractSprayProposalProvider {
    @Inject
    private IJvmTypeProvider.Factory      jvmTypeProviderFactory;
    @Inject
    private ITypesProposalProvider        typeProposalProvider;
    
    @Override
    public void completeJvmParameterizedTypeReference_Type(EObject model, Assignment assignment, ContentAssistContext context, ICompletionProposalAcceptor acceptor) {
        if (EcoreUtil2.getContainerOfType(model, CustomBehavior.class) != null) {
            final IJvmTypeProvider jvmTypeProvider = jvmTypeProviderFactory.createTypeProvider(model.eResource().getResourceSet());
            // Graphiti specific
            final JvmType interfaceToImplement = jvmTypeProvider.findTypeByName(ICustomFeature.class.getName());
            typeProposalProvider.createSubTypeProposals(interfaceToImplement, this, context, SprayPackage.Literals.BEHAVIOR__REALIZED_BY, TypeMatchFilters.canInstantiate(), acceptor);
        } else {
            super.completeJvmParameterizedTypeReference_Type(model, assignment, context, acceptor);
        }
    }

Now you will get the desired subtypes as proposals:

Note that this does only limit proposals to certain types. It does not prevent you from actually referring to another type that you don’t get proposed. This needs to be restricted in your scope provider.
 

Xtext 2.2 finally brings Maven support for Xtend

I cannot tell how often I was asked since introduction of Xtend2 how to compile them in a Maven build. This was just not possible until now due to the problem that to load and compile an Xtend class it is necessary to compile all of the Java classes that the Xtend class on before, and Java classes might depend on Xtend classes to be translated in order to be compilable.

Xtext 2.2.0 was just released yesterday, and for me the most important new feature is direct support through the xtend-maven-plugin plugin. I could not hesitate to test this feature, and created a small example.

Scenario

I have set up a small Maven project with 2 Java classes and 1 Xtend class that depend on each other.

To translate XtendClass1 to Java code and compile it requires that JavaClass1 is compiled before, and to compile JavaClass2 it is necessary that XtendClass is translated to Java code before.

Maven POM Configuration

I have decided to not use Maven Tycho for this example, and use the typical structure of an Eclipse project with usage of Xtend, i.e. sources in folder /src (instead /src/main/java) and xtend-gen to generate the Java code for Xtend classes to.

Repositories

Since shortly Xtext artifacts are available on maven.eclipse.org. The new Maven plugin is available on http://build.eclipse.org/common/xtend/maven/

	<repositories>
		<repository>
			<id>maven.eclipse.org</id>
			<url>http://maven.eclipse.org/nexus/content/groups/public/</url>
		</repository>
		<repository>
			<id>xtend</id>
			<url>http://build.eclipse.org/common/xtend/maven/</url>
		</repository>
	</repositories>
	<pluginRepositories>
		<pluginRepository>
			<id>xtend</id>
			<url>http://build.eclipse.org/common/xtend/maven/</url>
		</pluginRepository>
		<pluginRepository>
			<id>fornax</id>
			<url>http://fornax-platform.org/nexus/content/groups/public/</url>
		</pluginRepository>
	</pluginRepositories>

Dependencies

This small project setup requires just a mimimal set of dependencies to be configured:

	<dependencies>
		<dependency>
			<groupId>org.eclipse.xtend2</groupId>
			<artifactId>org.eclipse.xtend2.lib</artifactId>
			<version>2.2.0</version>
		</dependency>
		<dependency>
			<groupId>org.eclipse.xtext</groupId>
			<artifactId>org.eclipse.xtext.xtend2.lib</artifactId>
			<version>2.2.0.v201112061305</version>
		</dependency>
		<dependency>
			<groupId>com.google.inject</groupId>
			<artifactId>com.google.inject</artifactId>
			<version>2.0.0.v201105231817</version>
		</dependency>
	</dependencies>

Source folders

The main source folder src/ can be configured with the build/sourceDirectory setting, for xtend-gen we need the support of the build-helper-maven-plugin.

	<build>
		<sourceDirectory>src</sourceDirectory>
		<plugins>
			<plugin>
				<groupId>org.codehaus.mojo</groupId>
				<artifactId>build-helper-maven-plugin</artifactId>
				<version>1.7</version>
				<executions>
					<execution>
						<id>add-source</id>
						<phase>generate-sources</phase>
						<goals>
							<goal>add-source</goal>
						</goals>
						<configuration>
							<sources>
								<source>xtend-gen</source>
							</sources>
						</configuration>
					</execution>
				</executions>
			</plugin>
			...

The xtend-gen folder must be emptied when executing mvn clean, this requires some additional configuration of the maven-clean-plugin.

			<plugin>
				<artifactId>maven-clean-plugin</artifactId>
				<version>2.4.1</version>
				<configuration>
					<filesets>
						<fileset>
							<directory>xtend-gen</directory>
							<includes>
								<include>**</include>
							</includes>
						</fileset>
					</filesets>
				</configuration>
			</plugin>

xtend-maven-plugin

Now let’s finally come to the Xtend plugin. The plugin is configured as follows:

			<plugin>
				<groupId>org.eclipse.xtend2</groupId>
				<artifactId>xtend-maven-plugin</artifactId>
				<version>2.2.2</version>
				<executions>
					<execution>
						<goals>
							<goal>compile</goal>
							<goal>testCompile</goal>
						</goals>
						<configuration>
							<outputDirectory>xtend-gen</outputDirectory>
						</configuration>
					</execution>
				</executions>
			</plugin>

The outputDirectory is optional, but leaving it out the sources are generated to src/main/xtend-gen, and we want to generate to xtend-gen in the project root.

The plugin does not define a lifecycle mapping for M2E, which leads to an error marker “Plugin execution not covered by lifecycle configuration”. This is a well known issue when using M2E, and requires some additional work on the plugin. I have opened Bug#366118 for this.

Sample Project

The project described above is shared on Github: https://github.com/kthoms/xtext-experimental

In Eclipse the project looks like this after import (assuming the M2E plugin installed):

Build Execution

The classes in the project can now be built using mvn clean install. This will produce the following output:

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building xtend-maven-classic 2.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ xtend-maven-classic ---
[INFO] Deleting /Users/thoms/git/xtext-experimental/maven/xtend.maven/target
[INFO] Deleting /Users/thoms/git/xtext-experimental/maven/xtend.maven/xtend-gen (includes = [**], excludes = [])
[INFO] 
[INFO] --- build-helper-maven-plugin:1.7:add-source (add-source) @ xtend-maven-classic ---
[INFO] Source directory: /Users/thoms/git/xtext-experimental/maven/xtend.maven/xtend-gen added.
[INFO] 
[INFO] --- xtend-maven-plugin:2.2.0:compile (default) @ xtend-maven-classic ---
[WARNING] 
WARNING: 	XtendClass1.xtend - /Users/thoms/git/xtext-experimental/maven/xtend.maven/src/mypackage/XtendClass1.xtend
7: The value of the field XtendClass1.cls is not used
[INFO] Compiling 1 source file to xtend-gen
[INFO] 
[INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ xtend-maven-classic ---
[WARNING] Using platform encoding (MacRoman actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/thoms/git/xtext-experimental/maven/xtend.maven/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ xtend-maven-classic ---
[WARNING] File encoding has not been set, using platform encoding MacRoman, i.e. build is platform dependent!
[INFO] Compiling 3 source files to /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.923s
[INFO] Finished at: Thu Dec 08 23:01:42 CET 2011
[INFO] Final Memory: 20M/81M
[INFO] ------------------------------------------------------------------------

From the output it can be seen that at the end 3 Java classes are compiled, after the xtend-maven-plugin created the Java code for the Xtend class.

Enabling debug output with -X reveals some more insight, how the plugin works:

[DEBUG] Configuring mojo 'org.eclipse.xtend2:xtend-maven-plugin:2.2.0:compile' with basic configurator -->
[DEBUG]   (f) outputDirectory = xtend-gen
[DEBUG]   (f) project = MavenProject: org.eclipse.xtext.example:xtend-maven-classic:2.2.0-SNAPSHOT @ /Users/thoms/git/xtext-experimental/maven/xtend.maven/pom.xml
[DEBUG]   (f) tempDirectory = /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend
[DEBUG] -- end configuration --
[DEBUG] load xtend file 'file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/mypackage/XtendClass1.xtend'
[DEBUG] Parsing took: 42 ms
...
[DEBUG] create java stub 'mypackage/XtendClass1.java'
[DEBUG] invoke batch compiler with '-cp /Users/thoms/git/xtext-experimental/maven/xtend.maven/src:/Users/thoms/.m2/repository/org/eclipse/xtend2/org.eclipse.xtend2.lib/2.2.0/org.eclipse.xtend2.lib-2.2.0.jar:/Users/thoms/.m2/repository/com/google/guava/guava/10.0.1/guava-10.0.1.jar:/Users/thoms/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xtend2.lib/2.2.0.v201112061305/org.eclipse.xtext.xtend2.lib-2.2.0.v201112061305.jar:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xbase.lib/2.2.0.v201112061305/org.eclipse.xtext.xbase.lib-2.2.0.v201112061305.jar:/Users/thoms/.m2/repository/com/google/inject/com.google.inject/2.0.0.v201105231817/com.google.inject-2.0.0.v201105231817.jar -d /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/classes -1.5 -proceedOnError /Users/thoms/git/xtext-experimental/maven/xtend.maven/src /Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/stubs'
[DEBUG] classpath used for Xtend compilation : [file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/, file:/Users/thoms/.m2/repository/org/eclipse/xtend2/org.eclipse.xtend2.lib/2.2.0/org.eclipse.xtend2.lib-2.2.0.jar, file:/Users/thoms/.m2/repository/com/google/guava/guava/10.0.1/guava-10.0.1.jar, file:/Users/thoms/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar, file:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xtend2.lib/2.2.0.v201112061305/org.eclipse.xtext.xtend2.lib-2.2.0.v201112061305.jar, file:/Users/thoms/.m2/repository/org/eclipse/xtext/org.eclipse.xtext.xbase.lib/2.2.0.v201112061305/org.eclipse.xtext.xbase.lib-2.2.0.v201112061305.jar, file:/Users/thoms/.m2/repository/com/google/inject/com.google.inject/2.0.0.v201105231817/com.google.inject-2.0.0.v201105231817.jar, file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/src/, file:/Users/thoms/git/xtext-experimental/maven/xtend.maven/target/xtend/classes/]

The plugin invokes the Xtend Batch compiler through a spawned JVM. The batch compiler is realized in class org.eclipse.xtext.xtend2.compiler.batch.Xtend2BatchCompiler from the newly added plugin org.eclipse.xtext.xtend2.standalone.

After the build XtendClass1.java is generated as expected to xtend-gen. When opening the target folder it can be seen that the plugin produces a Java stub class to target/stubs for the Xtend class. Finally, everything is compiled and the project is clean

Limitation

The plugin only takes one source folder into account, namely the one configured by build/sourceDirectory. This is a problem when you have multiple source folders, which is quite typical for Xtext projects, namely src and src-gen. This is reported as , which is fixed in the meantime. I have deployed a patched version as unofficial release 2.2.2.

Conclusion

I deeply desired this plugin and this finally allows that the xtend-gen folder does not need to be checked in. The sample used here was just simple. Next would be a real life project were I want to apply the plugin, most likely in Project Spray. This project uses Xtend based code generation heavily and Maven Tycho for the build. The project has not been upgraded to Xtext 2.2.0 (of course, it was just released), and I guess we have to upgrade to Xtext 2.2.0 before we can use the plugin. But if this allows us to finally remove the xtend-gen folders from the repository this alone is worth upgrading.

Nice work, Xtext team!

Xtext 2.1.1 artifacts available on Eclipse Maven repository

Until now it was always a struggle to make Xtext (or Eclipse) artifacts available on a public available repository. It seems that this has now finally found a good end.

Today Achim Demelt pointed me to a Google+ posting from Dennis Huebner (our Xtext build manager) where he announced that Xtext artifacts are now automatically available on the Nexus repository at maven.eclipse.org.

The given POMs have all dependencies from their bundle manifests. This is the recommended “Maven way”, but you may face that too many dependencies are pulled transitive, and you may have to cut them down by explicit exclusions. I haven’t tested this with the new repository, but this is what I experienced when doing a similar approach earlier with Xtext 2.0.1, whose artifacts are deployed on the Fornax Repository. Use the dependency:tree plugin, or better M2E, to identify possible exclusions.

EclipseCon 2011 warmup

EclipseCon is just about to start and I am glad that I can make it this year. I have the chance to present Project Spray on thursday (Nov 3rd, 11:30 AM Bürgersaal 2) together with Jos Warmer. Spray aims to provide tooling to facilitate the creation of visual editors based on the Graphiti framework by the use of Xtext based DSL and Xtend code generator. You can basically compare it with what GMF Tools does for GMF, just with a DSL based approach.

The project was founded at this year’s Code Generation conference, and Jos contributed an initial state of Spray derived from a customer’s project back in August. Since then the project team refactored the code base quite a bit. We are still in an early project phase. Most of the development is done in spare time yet, like Open Source projects often start. That was quite tough besides my full packed work schedule. Many hours on train or evenings in the hotel were spent to push this project.

At the moment I am just about to release version 0.3.0 of the Spray SDK, which I will use for our demo on thursday. Of course we want to gain a large audience. I am honest enough to say that attendees won’t see a production ready tool yet, but of course it is in working state. Jan announced a pie fight for visual tooling, I am ready to open my mouth wide enough to catch the incoming pies.

For our session we don’t want to start any discussion on graphical editing vs. visualization, whether GMF is better than Graphiti, or DSL / code generation vs. framework approach. There is always a niche for everything, and we are confident that Spray fills one of these gaps.

Besides what an end user actually can do with Spray at this moment there are quite some hidden gems in the project. I am using Spray to have also a non-trivial example for Eclipse Modeling tooling. You will find, for example

• a Maven Tycho based build for Xtext 2 based projects
• SWTBot based UI tests, integrated in the Maven build
• a project wizard with Xtend based code generation
• an advanced example use of Xbase in a DSL
• documentation based on WikiText, integrated in the build
• unit testing of the Xtext DSL with xtext-utils unittesting
• separation of the Xtend code generator from the language project

At EclipseCon I will take the chance to exchange with some experts in the fields of Modeling, especially Xtext and Graphiti. I will lay another focus on Build systems, especially Maven. Besides that there are many other interesting talks, far more than I can attend. And not to forget to meet all the other Eclipse enthusiasts. See you there!