One Xpand user less, one happy Xtend user more

This week I am consulting a customer who had introduced a model based development approach based almost 10 years ago and used it with success since then. At the time back then Xpand was the most powerful code generation engine and UML models were often used to generate code from. Xtext did not even exist at that time. The customer uses Enterprise Architect and the Enterprise Architect exporter from the components4oaw. The last release was in 2011 and the project has not been developed further. For my customer the component just did what it should, so there was no direct need to change anything at the process. The EA exporter has its flaws, especially since it needs an Enterprise Architect installation and this means it only works on Windows machines. For Enterprise Architect users who do model based development, we therefore offer the YAKINDU Enterprise Architect Bridge, which scales better and can process .eap models on any platform.

My task was to help the customer to modernize their tool chain. To be fair, Xpand is not the right choice anymore. The Xtend Language combines the strengths from Xpand (great templating support, functional programming, static typing, polymorphic dispatch, mature Eclipse tooling) and resolved some weaknesses (performance, compiled code instead of interpreted, Java integration, extensibility of expressions).

For the customer who never had used Xtend so far, but was quite familiar with Xpand, it was quite a surprise how close both languages really are. Most of the concepts can be mapped 1:1 from good old Xpand to Xtend. We created a small generator from scratch and copied functions and templates and translated them manually for demonstration. They understood Xtend within minutes then. Most of the work is monkey-see-monkey-do. For such cases I wrote a small migration script which can translate Xpand,Xtend(1) and Check code to Xtend(2) classes. We used that script now for an initial translation.

One of the reasons why this migration script is not published is that cannot translate Xpand code completely to Xtend. The tool parses Xpand templates and traverses the AST to transform the expressions to Xtend equivalents. But it does not know about the type system which is used by the generator.

And here Xtend is not as powerful as Xpand, especially when using UML. In Xpand, the UML type system adapter analyzed the applied profiles of a model to create virtual types for stereotypes. Elements with stereotypes applied can be processed as if they were of a subtype of the extended type, and tagged values became attributes. In Xtend, there is only the Java type system, and for processing UML models this means that templates have to use the UML metamodel directly. The old “feeling” of a real type system can be simulated with a set of extension functions. I usually introduce an extension class per profile which offers for example a method per tagged value. The creation of such an extension class can again be automated by generating it from a profile .uml model.

Another disadvantage against the Xpand framework is that Xtend is not a code generation framework, but a general purpose programming language. How a generator component looks like that invokes the templates, where to produce output to, how to integrate constraint checks, this all is not provided.

What I usually do here is to use Xtext’s infrastructure like the IGenerator interface and validation based on Xtext’s AbstractDeclarativeValidator and @Check annotations. I reuse Xtext’s MWE Reader and Generator component. However, this requires some work and advanced knowledge. The basic approach is here to make UML models recognized by Xtext as a generic EMF resource. To actually use the Xtext components, a generator specific Guice module has to be created which extends AbstractGenericResourceRuntimeModule and satisfy several dependencies which an Xtext language already as configured by default. A good part of the approach was described by Christian Dietrich in his blog posts “Xtend2 Code Generators with Non-Xtext Models” and “Xtext 2.0 and UML“. I may go into details in a later blog post.

Although Xtend is such a natural choice for writing code generators and it is so well-integrated in the Xtext ecosystem, there is framework support missing for non-Xtext models. It is easy to write a trivial framework, but why should everyone start writing their own when good infrastructure already exists in the Xtext framework? From what I experienced again, there is need to have some more framework support for this use case. I doubt that it could be part of Xtext itself, but maybe we will provide a framework for Xtend based code generators in the future.

At the end we were able to show and demonstrate a migration path in one single workshop day. The customer was happy to save several days or weeks of time. The workshop costs were compensated by far for them. From a sales perspective it might not be wise to leave a customer in a state where he is not dependent on our services in the next time, but this is not how we work. For me it feels right.

Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime.

Fornax MWE Workflow Maven Plugin 3.5.1 published on Maven Central

The Fornax Workflow plugin is a Maven Plugin that executes MWE/MWE2 workflows within Maven. It has been there for quite some years now, and whoever needed to integrate MWE/MWE2 workflows in a headless build was likely using it. The Fornax Platform has been an address where components around openArchitectureWare, Xpand and Xtext have been developed. While all other subprojects don’t play any role anymore, the Workflow plugin is still in frequent use.

Over the years we had to change the underlying infrastructure some times. The plugin was hosted on the project’s own repository server, and projects using the plugin had to configure an additional plugin repository either in their POMs or settings.xml. This was undesired, but at the end not really a blocker. However, with a recent change of the repository manager, users experienced problems accessing the Fornax repository http://www.fornax-platform.org/nexus/content/groups/public. Currently, this URL is redirected to a server hosted at itemis, and users might get problems with the HTTPS connection.

It always bothered me that we had to host this plugin on a separate repository, and since it is a widely used component, it is logical that it should be available from the Central repository. But it was never a blocker for me. Now finally I got the driver to change this.

Long story short, the plugin is now published at Maven Central as version 3.5.1. I highly recommend to upgrade to this version and remove the Fornax Maven repository from your configuration. The coordinates did not change, they are still org.fornax.toolsupport:fornax-oaw-m2-plugin. I would like to change this sometime in the future (e.g. the name parts “oaw” and “m2” are not up-to-date anymore), maybe with moving development to another project hosting platform.

Version 3.5.1 does not differ much from 3.4.0, which is the version likely used by the world today. The main work was on refactoring the POM and its parent in order to meet the requirements for deployment on Maven Central. Further, I worked on automation of the release process with the maven-release-plugin.

There is one additional feature in 3.5.x: The new property useTestScope can be used to skip dependencies from the test scope for computation of the Java classpath used to execute a workflow in forked mode. On Windows systems the classpath sometimes reaches the limit of allowed command line length, especially since the local Maven repository is below the user’s home directory by default, which has already a rather long path prefix. By default, the plugin will exclude now these test scope dependencies unless the user configures the property explicitly. In 3.5.0 there was a small logical bug with this feature which made the plugin unusable, so please do not use that version. The version 3.5.1 can be used without problems for all using 3.4.0 so far.

Setting template file encoding in Xpand workflow

A typical problem when running Xpand is file encoding of the Xpand templates. Xpand uses guillemot brackets («»), which are not available in all character sets. Often, ISO-8859-1 and UTF-8 are good choices for encoding Xpand templates. However, on Mac the default encoding is MacRoman, and I would recommend to change that for interoperability reason. I would recommend changing the resource encoding setting on the root folder containing Xpand templates, or even the project, and checking in the settings. This way the encoding is shared amongst team members.

When executing the Xpand generator with MWE it is important that the templates are read with the right encoding. Otherwise you will get an error like this:
org.eclipse.internal.xtend.xtend.parser.ParseException: no viable alternative at character '�' on line 159
Xpand’s Generator component has a property fileEncoding, but this is the encoding used by the outlets to write files. If you need to change the behavior for reading files, you need to set the encoding value of the ResourceManager. The configuration is like this:

MWE1:

    ...

MWE2

  component = org.eclipse.xpand2.Generator {
    ...
    resourceManager = org.eclipse.xtend.expression.ResourceManagerDefaultImpl {
     	fileEncoding = "ISO-8859-1"
    }
  }

Using Xtext injected types in Xpand

During a project migration I stumbled over the need to access the IScopeProvider from an Xtext based language within Xtend(1) extensions that are used from Xpand templates. This is not quite how Xpand would work out-of-the-box, so here is an approach to solve it.

InjectableGenerator

I subclassed org.eclipse.xpand2.Generator and called it InjectableGenerator. The intention is to register a global variable for a configurable set of types (e.g. the IScopeProvider). The name of the variable is the class name that should be get from Guice. To add injectors, instances of ISetup are registered to the component, similar to Xtext’s org.eclipse.xtext.mwe.Reader component.

The hook that is used to register the global variables to Xpand’s execution context is the method getGlobalVars(WorkflowContext ctx), which is overridden here.

package org.eclipselabs.xtext.xpand;

import java.util.List;
import java.util.Map;
import java.util.Set;

import org.eclipse.emf.mwe.core.WorkflowContext;
import org.eclipse.xpand2.Generator;
import org.eclipse.xtend.expression.Variable;
import org.eclipse.xtext.ISetup;

import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import com.google.inject.ConfigurationException;
import com.google.inject.Injector;

/**
 * Allows Xtext based dependency injection. Injected instances are registered as
 * global vars to the Xpand Execution Context.
 */
public class InjectableGenerator extends Generator {
	private List<Injector> injectors = Lists.newArrayList();

	public void addRegister(ISetup setup) {
		injectors.add(setup.createInjectorAndDoEMFRegistration());
	}

	protected List<Injector> getInjectors() {
		return injectors;
	}
	
	private Set<Class<?>> injectedTypes = Sets.newHashSet();

	public void addInjectGlobalVar (String className) {
		try {
			Class<?> clazz = Class.forName(className);
			injectedTypes.add(clazz);
		} catch (Exception e) {
			throw new IllegalArgumentException("Invalid type: "+className);
		}
	}
	
	@Override
	protected Map<String, Variable> getGlobalVars(WorkflowContext ctx) {
		Map<String, Variable> result = super.getGlobalVars(ctx);
		// try to get an instance of each configured type
		for (Class<?> clazz : injectedTypes) {
			boolean configured = false;
			for (Injector injector : injectors) {
				try {
					Object obj = injector.getInstance(clazz);
					result.put(clazz.getName(), new Variable(clazz.getName(), obj));
					configured = true;
				} catch (ConfigurationException e) {
					; // ignore
				}
			}
			if (!configured) {
				throw new IllegalStateException("Could not configure instance of "+clazz.getName());
			}
		}
		return result;
	}
}

Workflow Configuration

In your workflow configuration you would of course now use the specialized Generator component. It could be used like this:

...
component = org.eclipselabs.xtext.xpand.InjectableGenerator {
   register = my.dsl.MyDslStandaloneSetup {}
   injectGlobalVar = "org.eclipse.xtext.scoping.IScopeProvider"
   // injectGlobalVar = ... other types
   
   // further configuration of Xpand Generator
}

Execution context aware Java extensions

It is not possible to directly access the ExecutionContext from within Xpand templates or Xtend functions. We need to delegate to Java extensions. The classes that we delegate to must implement the IExecutionContextAware interface. The setExecutionContext(ExecutionContext ctx) method is the right place to retrieve the IScopeProvider and bind it to an instance variable.

public class MyExtensions implements IExecutionContextAware {
	protected IScopeProvider scopeProvider;

	@Override
	public void setExecutionContext(ExecutionContext ctx) {
		if (ctx.getGlobalVariables().get("scopeProvider")!=null) {
			this.scopeProvider = (IScopeProvider) ctx.getGlobalVariables().get("scopeProvider").getValue();
		} else if (ctx.getGlobalVariables().get(IScopeProvider.class.getName())!=null) {
			this.scopeProvider = (IScopeProvider) ctx.getGlobalVariables().get(IScopeProvider.class.getName()).getValue();
		} else {
			throw new IllegalStateException("No IScopeProvider found as global var in the execution context");
		}
	}

	// add (non-static!) extension methods here!
        public List<EObject> someMethod (org.eclipse.emf.ecore.EObject context) {
          ... do something
        }
}

Note that we also query the global variable named “scopeProvider” here. This is because in another context (i.e. Check based validation) the same extension would be used with another execution context. The InjectableExecutionContext binds the IScopeProvider to that name.

It might be a good idea to have an abstract base class for these Extension classes.

Now you can use the methods defined in the Java extension class and access the required instances from your language configuration within.

// file: MyExtensions.ext
List[emf::EObject] someMethod (emf::EObject ctx) : JAVA MyExtensions.someMethod(org.eclipse.emf.ecore.EObject);

Xtext 2.0 / Xpand 1.1 plugins available as Maven artifacts

It’s now one week after release of Eclipse Indigo, and with that Xtext 2.0 and Xpand 1.1. For those who use Maven without Tycho to build projects using these frameworks it is of course necessary to have all related plugins available in some Maven repository.

With the help of the osgi-to-maven2 project I was now able with some tweaks to create the necessary POMs and deploy scripts for the plugins. What osgi-to-maven2 basically does is to analyze the plugin manifests and create POMs for them including dependencies. The project is not maintained at the moment and has a problem especially with the parsing of the plugins org.eclipse.ui.views and org.eclipse.ui.workbench, which have some strange directives in the Export-Package section where I could not find documentation about. The underlying manifest parser from Springsource (com.springsource.util.osgi) vomits when parsing those manifests, and also upgrading to the latest release does not help. Time to open a bug.


Export-Package: org.eclipse.ui;mandatory:="ui.workb
ench",org.eclipse.ui.about,org.eclipse.ui.actions;ui.workbench=split;
mandatory:="ui.workbench",org.eclipse.ui.activities,org.eclipse.ui.ap
plication,org.eclipse.ui.branding,org.eclipse.ui.browser;ui.workbench
=split;mandatory:="ui.workbench",org.eclipse.ui.commands,org.eclipse.

Also the osgi-to-maven2 tool does not handle absent optional libraries so far.

The Xtext 2.0, Xpand 1.1 and dependent plugins are available at the Fornax Platform’s repository http://fornax-platform.org/nexus/content/groups/public/ under the groupId org.eclipse.plugins.

Based on the deployed plugins it should be possible to setup a plain Maven build for Xtext 2 based projects.

Tycho build works also for oAW 4.3 projects

Yesterday I showed how Xtext 1.0 projects can be build with Maven Tycho. So why shouldn’t the same setup don’t work for ‘old’ openArchitectureWare Xpand projects? This works like a charm. I took the simple project that the Xpand project wizard creates and added the POM, and it worked! You can download the sample project here. See the POM below, a setup for other oAW Xpand projects should be rather similar.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>p2.osgi.bundle</groupId>
	<artifactId>org.openarchitectureware.example.xpand.tycho</artifactId>
	<version>1.0.0</version>
	<packaging>eclipse-plugin</packaging>
	<name>org.openarchitectureware.example.xpand.tycho</name>

	<properties>
		<version.tycho>0.9.0</version.tycho>
	</properties>

	<build>
		<resources>
			<resource>
				<directory>src</directory>
			</resource>
		</resources>
		<plugins>
			<plugin>
				<groupId>org.sonatype.tycho</groupId>
				<artifactId>tycho-maven-plugin</artifactId>
				<version>${version.tycho}</version>
				<extensions>true</extensions>
			</plugin>
			<plugin>
				<groupId>org.sonatype.tycho</groupId>
				<artifactId>target-platform-configuration</artifactId>
				<version>${version.tycho}</version>
				<configuration>
					<resolver>p2</resolver>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.fornax.toolsupport</groupId>
				<artifactId>fornax-oaw-m2-plugin</artifactId>
				<version>3.1.0-SNAPSHOT</version>
				<configuration>
					<workflowEngine>oaw</workflowEngine>
					<workflowDescriptor>workflow/generator.oaw</workflowDescriptor>
				</configuration>
				<executions>
					<execution>
						<phase>generate-sources</phase>
						<goals>
							<goal>run-workflow</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<artifactId>maven-clean-plugin</artifactId>
				<version>2.4.1</version>
				<configuration>
					<filesets>
						<fileset>
							<directory>src-gen</directory>
							<includes>
								<include>**/**</include>
							</includes>
						</fileset>
					</filesets>
				</configuration>
			</plugin>
		</plugins>
	</build>
	<repositories>

		<repository>
			<id>p2.openarchitectureware</id>
			<url>http://www.openarchitectureware.org/updatesite/milestone/</url>
			<layout>p2</layout>
		</repository>
		<repository>
			<id>p2.eclipse.helios</id>
			<url>http://download.eclipse.org/releases/helios</url>
			<layout>p2</layout>
		</repository>
		<!-- At the moment the Fornax plugin is only available as snapshot -->
		<repository>
			<id>fornax-snapshots</id>
			<url>http://www.fornax-platform.org/archiva/repository/snapshots/</url>
			<releases>
				<enabled>false</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</repository>
	</repositories>
	<pluginRepositories>
		<pluginRepository>
			<id>fornax-snapshots</id>
			<url>http://www.fornax-platform.org/archiva/repository/snapshots/</url>
			<releases>
				<enabled>false</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</pluginRepository>
	</pluginRepositories>
</project>

Lazy evaluation in Xpand

Usually code generation is a purely sequential process. Since the model does not change during the generation of an artifact all content can be computed in the template where it is needed for the output. But sometimes there is the wish to defer the output to a later point of time during the generation of an artifact.

The typical use case for this is import statements. If for example you want to generate a Java class and want to import all used types then the following alternatives are given:

• Compute the types that the about-to-be-generated class will use
• Print out all type names full qualified whenever needed and organize the imports with a postprocessor. For Java code generation the Hybridlabs Beautifier is used widely.

However, both approaches do not seamlessly solve the problem. What really is needed is some kind of lazy evaluation in Xpand. Therefore Jos Warmer wrote a feature proposal once. The feature that he proposed for Xpand is called Insertion Point. The idea was to mark some point in the Xpand template where some code will be inserted at a later point of time. Code is evaluated into this insertion point when the content can be derived easier.

From this feature proposal Feature Request #261607 was created in Eclipse Bugzilla system. In this bug entry, and also offline, a lively discussion arose in the team. The challenges for this feature request were:

• The Xpand language has a rather small set of keywords. Adding this feature, which is used in some cases only, should not introduce too much changes to the Xpand language
• An implementation should not break existing code
• The solution should enable decent evaluation, e.g. avoiding duplicates, sorting

The latest proposal just introduces just one new keyword in Xpand, but requires an implementation pattern with Xtend function. The proposal is to add a keyword ONFILECLOSE to the EXPAND statement. By calling EXPAND with ONFILECLOSE the evaluation of the EXPAND statement is deferred until the FILE statement is closed. Any state that is used by the called definition is computed during the FILE evaluation. The EXPAND statement has to be evaluated with the execution context which is active when reaching the EXPAND statement.

Let’s see this by example. As example we take the project that Xpand’s project wizard creates, with small changes. The entities and types have now an additional ‘packageName’ attribute, and the both entities have been assigned different packages ‘entities1’ and ‘entities2’. Additionally entity ‘Person’ has a feature ‘birthday’ of type Date, which is mapped to java.util.Date. Therefore class Person.java has to import entities2.Address and java.util.Date. The used types should be collected when rendering instance variables and accessor methods, but inserted earlier in the code.

First take a look at the template code:

As you can see the definition ImportBlock is called for each Type instance (e.g. Entity and DataType instances) in the collection returned by the Xtend function UsedType(). In an alternative approach this function would be responsible to compute all the types that will be used by this Entity instance. But for the new implementation it just creates an empty list and returns that one. The implementation of the UsedType() function (in file GeneratorExtensions.ext) is:

create List[Type] UsedType (Entity e) : (List[Type]) {};

So at the time the Xpand engine reaches EXPAND ImportBlock… the list would be empty. Now note the ONFILECLOSE keyword at the end of the EXPAND statement. This one tells the engine now that the evaluation of this Xpand statement should be deferred until the file is about to be closed. During evaluation of the template code, in the FOREACH loop, another extension function addUsedType(f.type) is called. This one adds the type of the current processed feature to the collection returned by UsedTypes(). Therfore it is important that the UsedType() function uses the create keyword, since we want to create the collection on first access for one Entity and return the same instance when UsedTypes() is called later for that Entity again.

The function addUsedType() is used in the template like this:

«addUsedType(f.type)»

Xpand would print out the result of the function as string, but we don’t want to produce any output by calling the function. Therefore we assure that the function adds the type to the collection and returns an empty string:

addUsedType (Entity e, Type t) : UsedType(e).add(t) -> "";

During evaluation it could be that types were used multiple times within the template, but we want just one import statement per type. Further we don’t need imports for types from the java.lang package (here the packageName information for the DataType instances String and Integer is null) or for types that are in the same package like the entity. Therefore we transform the UsedType() collection before finally invoking the ImportBlock definition.

«EXPAND ImportBlock FOREACH UsedType()
 .select(t|t.packageName!=null && t.packageName!=this.packageName).toSet()
 .sortBy(t|t.qualifiedName()) ONFILECLOSE»

Conclusion

Lazy evaluation allows code generated with Xpand in an non-sequential matter. The proposed solution using ONFILECLOSE solves the desired Insertion Point feature by adding just one additional keyword to the Xpand language. It does not break existing template code. When accepted this code will be contributed to Xpand 0.8.0-M6 soon. For those who want to test it I have created a feature patch for the org.eclipse.xpand feature. The example project with the sources listed in this article can be downloaded here.

Feedback is welcome and is best placed in the bugzilla feature request.