One Xpand user less, one happy Xtend user more

This week I am consulting a customer who had introduced a model based development approach based almost 10 years ago and used it with success since then. At the time back then Xpand was the most powerful code generation engine and UML models were often used to generate code from. Xtext did not even exist at that time. The customer uses Enterprise Architect and the Enterprise Architect exporter from the components4oaw. The last release was in 2011 and the project has not been developed further. For my customer the component just did what it should, so there was no direct need to change anything at the process. The EA exporter has its flaws, especially since it needs an Enterprise Architect installation and this means it only works on Windows machines. For Enterprise Architect users who do model based development, we therefore offer the YAKINDU Enterprise Architect Bridge, which scales better and can process .eap models on any platform.

My task was to help the customer to modernize their tool chain. To be fair, Xpand is not the right choice anymore. The Xtend Language combines the strengths from Xpand (great templating support, functional programming, static typing, polymorphic dispatch, mature Eclipse tooling) and resolved some weaknesses (performance, compiled code instead of interpreted, Java integration, extensibility of expressions).

For the customer who never had used Xtend so far, but was quite familiar with Xpand, it was quite a surprise how close both languages really are. Most of the concepts can be mapped 1:1 from good old Xpand to Xtend. We created a small generator from scratch and copied functions and templates and translated them manually for demonstration. They understood Xtend within minutes then. Most of the work is monkey-see-monkey-do. For such cases I wrote a small migration script which can translate Xpand,Xtend(1) and Check code to Xtend(2) classes. We used that script now for an initial translation.

One of the reasons why this migration script is not published is that cannot translate Xpand code completely to Xtend. The tool parses Xpand templates and traverses the AST to transform the expressions to Xtend equivalents. But it does not know about the type system which is used by the generator.

And here Xtend is not as powerful as Xpand, especially when using UML. In Xpand, the UML type system adapter analyzed the applied profiles of a model to create virtual types for stereotypes. Elements with stereotypes applied can be processed as if they were of a subtype of the extended type, and tagged values became attributes. In Xtend, there is only the Java type system, and for processing UML models this means that templates have to use the UML metamodel directly. The old “feeling” of a real type system can be simulated with a set of extension functions. I usually introduce an extension class per profile which offers for example a method per tagged value. The creation of such an extension class can again be automated by generating it from a profile .uml model.

Another disadvantage against the Xpand framework is that Xtend is not a code generation framework, but a general purpose programming language. How a generator component looks like that invokes the templates, where to produce output to, how to integrate constraint checks, this all is not provided.

What I usually do here is to use Xtext’s infrastructure like the IGenerator interface and validation based on Xtext’s AbstractDeclarativeValidator and @Check annotations. I reuse Xtext’s MWE Reader and Generator component. However, this requires some work and advanced knowledge. The basic approach is here to make UML models recognized by Xtext as a generic EMF resource. To actually use the Xtext components, a generator specific Guice module has to be created which extends AbstractGenericResourceRuntimeModule and satisfy several dependencies which an Xtext language already as configured by default. A good part of the approach was described by Christian Dietrich in his blog posts “Xtend2 Code Generators with Non-Xtext Models” and “Xtext 2.0 and UML“. I may go into details in a later blog post.

Although Xtend is such a natural choice for writing code generators and it is so well-integrated in the Xtext ecosystem, there is framework support missing for non-Xtext models. It is easy to write a trivial framework, but why should everyone start writing their own when good infrastructure already exists in the Xtext framework? From what I experienced again, there is need to have some more framework support for this use case. I doubt that it could be part of Xtext itself, but maybe we will provide a framework for Xtend based code generators in the future.

At the end we were able to show and demonstrate a migration path in one single workshop day. The customer was happy to save several days or weeks of time. The workshop costs were compensated by far for them. From a sales perspective it might not be wise to leave a customer in a state where he is not dependent on our services in the next time, but this is not how we work. For me it feels right.

Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime.

Xbase Customization: Redefining operator keywords

If you use Xbase in your Xtext based DSL, you are usually satisfied with the set of operators the expression language defines. They are closely related to what you are used to in Java or similar languages.

However, in a customer’s workshop the customer wished to have custom keywords for some operators. For example, the operator && should be alternatively presented as AND, and the || operator as OR.

To demonstrate this customization we’ll start with Xtext’s famous Domainmodel Example. The Domainmodel.xtext grammar is derived from org.eclipse.xtext.xbase.Xbase and the Operation rule uses an XBlockExpression for the Operation’s body:

grammar org.eclipse.xtext.example.domainmodel.Domainmodel with org.eclipse.xtext.xbase.Xbase

...

Operation:
	'op' name=ValidID '(' (params+=FullJvmFormalParameter (',' params+=FullJvmFormalParameter)*)? ')' (':' type=JvmTypeReference)?
		body=XBlockExpression;

Unit Test

We’ll extend the language test-driven, thus we first create a unit test that uses the new feature, but will fail first until we have successfully implemented it. Fortunately there is already a suitable test class in project org.eclipse.xtext.example.domainmodel.tests.

We extend the class ParserTest.xtend:

	@Test
	def void testOverriddenKeyword() {
		val model = '''
			package example {
			  entity MyEntity {
			    property : String
			    op foo(String s) {
			    	return s!= null && s.length > 0 AND s.startsWith("bar")
			    }
			  }
			}
		'''.parse
		val pack = model.elements.head as PackageDeclaration
		val entity = pack.elements.head as Entity
		val op = entity.features.last as Operation
		val method = op.jvmElements.head as JvmOperation
		model.eResource.assertNoErrors
		Assert::assertEquals("boolean", method.returnType.simpleName)
	}

Note the Operation body, the expression uses both presentations of the And-operator.

return s!= null && s.length > 0 AND s.startsWith("bar")

When the ParserTest is executed it will now fail, of course, but only for the AND keyword:

java.lang.AssertionError: Expected no errors, but got :
ERROR (org.eclipse.xtext.diagnostics.Diagnostic.Linking) 
'The method or field AND is undefined' 
on XFeatureCall, offset 119, length 3
ERROR (org.eclipse.xtext.xbase.validation.IssueCodes.unreachable_code) 'Unreachable expression.' on XFeatureCall, offset 119, length 3

	at org.junit.Assert.fail(Assert.java:88)
	at org.eclipse.xtext.junit4.validation.ValidationTestHelper.assertNoErrors(ValidationTestHelper.java:187)
	at org.eclipse.xtext.example.domainmodel.tests.ParserTest.testOverriddenKeyword(ParserTest.java:268)
...

Overloading the operator rules

Looking at Xbase.xtext shows that Xbase defines separate data type rules for operators:

OpOr:
	'||';
OpAnd:
	'&&';

Xtext’s Grammar Mixin feature allows a redefinition of those rules. The obvious customization is in Domainmodel.xtext:

OpOr:
	'||' | 'OR';
OpAnd:
	'&&' | 'AND';

Regenerating the language’s Xtext implementation makes those keywords available to the syntax. However, the unit test still fails, but now with a different error:

java.lang.AssertionError: Expected no errors, but got :
ERROR (org.eclipse.xtext.diagnostics.Diagnostic.Linking) 
'AND cannot be resolved.' on XBinaryOperation, offset 119, length 3

	at org.junit.Assert.fail(Assert.java:88)
	at org.eclipse.xtext.junit4.validation.ValidationTestHelper.assertNoErrors(ValidationTestHelper.java:187)
	at org.eclipse.xtext.example.domainmodel.tests.ParserTest.testOverriddenKeyword(ParserTest.java:268)

OperatorMapping

The missing piece is a customization of class OperatorMapping. Thus we create a subclass OperatorMappingCustom with constant QualifiedNames for the additional operator keywords and bind it in DomainmodelRuntimeModule:

@Singleton
public class OperatorMappingCustom extends OperatorMapping {
	public static final QualifiedName AND_2 = create("AND");
	public static final QualifiedName OR_2 = create("OR");
}
public class DomainmodelRuntimeModule extends AbstractDomainmodelRuntimeModule {
	[...]
	public Class bindOperatorMapping() {
		return OperatorMappingCustom.class;
	}
}

A naive approach is here to overload the initializeMapping(), as the Javadoc suggests (“Clients may want to override #initializeMapping() to add other operators.“):

But this fails again:

com.google.inject.CreationException: Guice creation errors:

1) Error injecting constructor, java.lang.IllegalArgumentException: value already present: operator_and
  at org.eclipse.xtext.example.domainmodel.OperatorMappingCustom.(Unknown Source)
  at org.eclipse.xtext.example.domainmodel.OperatorMappingCustom.class(Unknown Source)
  while locating org.eclipse.xtext.example.domainmodel.OperatorMappingCustom
  while locating org.eclipse.xtext.xbase.scoping.featurecalls.OperatorMapping
    for field at org.eclipse.xtext.xbase.util.XExpressionHelper.operatorMapping(Unknown Source)
  while locating org.eclipse.xtext.xbase.util.XExpressionHelper
    for field at org.eclipse.xtext.xbase.validation.XbaseValidator.expressionHelper(Unknown Source)
  at org.eclipse.xtext.service.MethodBasedModule.configure(MethodBasedModule.java:57)
  while locating org.eclipse.xtext.example.domainmodel.validation.DomainmodelJavaValidator
Caused by: java.lang.IllegalArgumentException: value already present: operator_and
	at com.google.common.collect.HashBiMap.put(HashBiMap.java:237)
	at com.google.common.collect.HashBiMap.put(HashBiMap.java:214)
	at org.eclipse.xtext.example.domainmodel.OperatorMappingCustom.initializeMapping(OperatorMappingCustom.java:25)
	at org.eclipse.xtext.xbase.scoping.featurecalls.OperatorMapping.(OperatorMapping.java:121)
	at org.eclipse.xtext.example.domainmodel.OperatorMappingCustom.(OperatorMappingCustom.java:18)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	...

Instead, the method getMethodName() must be overridden and delegate the invocations of the new operators to the existing ones. The resulting OperatorMappingCustom is then:

@Singleton
public class OperatorMappingCustom extends OperatorMapping {
	public static final QualifiedName AND_2 = create("AND");
	public static final QualifiedName OR_2 = create("OR");

	@Override
	public QualifiedName getMethodName(QualifiedName operator) {
		if (AND_2.equals(operator)) {
			return getMethodName(AND);
		}
		if (OR_2.equals(operator)) {
			return getMethodName(OR);
		}
		return super.getMethodName(operator);
	}
}

Finally, the unit test will execute successful.

Summary

Xbase allows customization of operators by overriding the operator’s data type rule from Xbase.xtext. This adds the operator keywords to the language, but fails at runtime. Additionaly the class OperatorMapping must be customized and method getMethodName() overloaded.

Preparations for CodeGeneration 2013

As I am addicted to code generation and DSLs, the CodeGeneration conference in Cambridge is always a must each year. Last year I could not make it, since I had the chance to speak at EclipseCon North America, which was in the same week. This year Mark took EclipseCon into his considerations (it was last week), so me and my colleagues from itemis will be there again. Actually, this year we will be more itemis guys then ever. Mark already assumed in his opening words at CG2011 that almost everyone from itemis would be there, this year we prove itemis is larger. I think our company is so close related to the conference theme that it is natural that we have lots to present, and much interest to hear others about what they are doing and have learned in the past.

screenshot 2013-04-05 um 10.55.19

Before the actual CodeGeneration conference starts on Wednesday, there are some pre-conference activities. My colleagues Holger Schill and Moritz Eysholdt will hold an intensive 2-day Xtext workshop on monday and tuesday.

I will arrive monday noon, since I take the early flight directly from my hometown Dortmund to London Luton. From there, I have to take a 2 hour bus trip to Cambridge. In the evening, I plan to meet Holger, Moritz, Meinte Boersma and hopefully some others in the Castle Inn pub. When you arrive on monday, drop into the Castle Inn roughly at 20 PM (I guess we go to a restaurant before). You can reach me there on my mobile phone: screenshot 2013-04-05 um 12.30.02

DSC00412

On tuesday this year’s Language Workbench Challenge summit takes place. We have 14 submissions (wow!) for the LWC13 assignment. I have been working on the Xtext submission together with 2 colleagues, Johannes Dicks and Thomas Kutz. The results are available as open-source project lwc13-xtext at Eclipselabs. We have prepared a detailed step-by-step tutorial as submission paper. The resulting document LWC13-XtextSubmission.pdf is available for download. On the project homepage I have placed today a quick start tutorial. Oh boy, this project did cost some time. The actual solution is not much code, but as often, it is harder to write less code than more. It could be even less, but we took care of that the code is readable and understandable. And writing the document is at least that much work as the implementation.

screenshot 2013-04-08 um 13.52.06

Every presenter has only 15 minutes to present their approach. 15 minutes presentation for that much work. I guess that the other participants did invest also quite some time. Both of my co-authors got the chance to visit the conference, and Thomas will support me with the presentation. He will demo the resulting JSF application and DSL source code while I do the main talking. We did a test run of the talk yesterday evening, and easily exceeded 20 minutes. I think Angelo will bring his egg timer again, which begs no pardon with the talk time of speakers. But only that way we will be able to run 14 talks on one day. We will have to restrict on the most important aspects only.

Besides my colleagues Thomas and Johannes also Sven Lange will join us then. I have the pleasure to work with him in my long-time project at Deutsche Börse (German Stock Exchange), now since over half a year. Sven is a highly motivated, skilled and smart person. It is still the same project I have reported about at CodeGeneration 2009 together with my former colleague Heiko Behrens. Sven is full-time working on this project, while I am for 20% scheduled. We have migrated here a huge code generator project from Xpand to Xtend. This alone would be worth an experience report session. Sven is working on Xtend support for IntelliJ, which he might present in a Lightning Talk on wednesday.

Wednesday the conference will start. I will have my main talk “Alive and Kicking with Code Generation” together with Dr. Boris Baginski from ATOSS Software AG after lunch at 13:45.

screenshot 2013-04-05 um 10.57.34

Currently we are finalizing our presentation slides. Boris has been ill for some days and busy with a new release of their ASES product, a workforce management suite. This is really an interesting customer and project. They are evolving this product now for 25 years, and they make use of code generation for ages. I think one can say that it helped them to survive in their business, some contestants did not manage to make larger platform shifts and died. Most of them tried a big-bang replacement, but the business is too fast evolving so that the target is moving steadily. Boris and I will speak about this product and how it has been evolved over the years. ATOSS was one of the first major projects using openArchitectureWare 4 (which mainly means Xpand), and now they are currently preparing a shift to Xtend.

I am glad that this talk is already on wednesday, I never come to rest until I finished some talk. After it, I can just relax and enjoy the conference. I am expecting some interesting insights on different approaches. Especially experience reports are interesting for me. I did not finally decide which sessions I will attend. At the moment I plan to see John Hutchinson with “The Use of Model-Driven Development in Industry” in the morning, and Darius Silingas with “Why MDA Fails: Analysis of Unsuccessful Cases” in the afternoon.

In the evening it is again time for the punting boat tour. I already attended three times, but it will be great fun again for sure. Let’s hope the weather is not too bad. I saw a prediction of ~10°C and possibility of light shower. In the past we had luck, and on a warm, sunny day the tour is double fun. However, I’ll better put an umbrella into the suitcase.

On thursday I have again an active part in the hands-on session “Have Your Language Built While You Wait”, which is hosted by Risto Pohjonen from MetaCase. The idea of this session is that attendees can get a DSL with the language workbench of their choice built with the help of experts for this workbench. Of course I will assist on Xtext. If you had no chance to visit the Xtext workshop this might be your chance to get some hands on Xtext. This session was already run last year successfully. Last year my colleague Benjamin Schwertfeger took over the Xtext part, since we were at EclipseCon.

There are also some other talks around Xtext and Xtend. Both have been released in version 2.4 on March 20th, which brings some interesting new features. Most notably in regard to code generation are the Active Annotations. I guess this is also part of what Sven Efftinge will adress as future of code generation in his keynote “The Past, Present and Future of Code Generation” wednesday morning. More details he will present together with Sebastian Zarnekow in the tutorial “Internal DSLs with Xtend” (thursday 10:45-12:00). The last Xtext related talk will be from Moritz Eysholdt, called “Executable Specifications for Xtext Languages” (friday 10:45-12:15). I am actually not sure which of these talks I will attend personally. They are most relevant for my work, and I don’t work close enough with them to catch everything new in Xtext on my own. Thus, I’ll definetely would learn important aspects. On the other side, there are also other interesting talks in parallel.

The coming week will be an intensive experience with lots to learn and interesting persons to meet. Although I will really enjoy this time, I will be glad when I finally come back home. At the moment, my family is ill and I hope that I get not infected these days. I have been looking forward and worked for this event, so I am crossing fingers when I can board monday morning healthy.

I am sure the organizing team around Mark and Jacqui will do again a great job.

DSC00363

See you there and let’s make this event special!

Xtext Content Assist: Filtering keyword proposals

Sometimes when I introduce template proposals in Xtext editors I want them to be a replacement for a keyword proposal. By default you would get a proposal for the template AND for the keyword. In the following screenshot a template for a “class” concept exists, and a proposal for the “class” keyword, which would be inserted as part of the template at the cursor position.

The key to the solution is to overwrite the completeKeyword() method from your proposal provider class, e.g. like this:

public class SprayProposalProvider extends AbstractSprayProposalProvider {
    private static final Set<String>      FILTERED_KEYWORDS = Sets.newHashSet("text", "line", "class", "behavior", "style", "custom");
    @Override
    public void completeKeyword(Keyword keyword, ContentAssistContext contentAssistContext, ICompletionProposalAcceptor acceptor) {
        if (FILTERED_KEYWORDS.contains(keyword.getValue())) {
            // don't propose keyword
            return;
        }
        super.completeKeyword(keyword, contentAssistContext, acceptor);
    }
}

Christian Dietrich added in a comment how to filter a keyword that may occur more than once in your grammar. If you want to filter specific occurances you need to use the IGrammarAccess of your language:

@Inject MyDslGrammarAccess grammarAccess;

@Override
public void completeKeyword(Keyword keyword,ContentAssistContext contentAssistContext,ICompletionProposalAcceptor acceptor) {
  if (!grammarAccess.getGreetingAccess().getHelloKeyword_0().equals(keyword)) {
    super.completeKeyword(keyword, contentAssistContext, acceptor);
  }
}

xtext-utils unittesting 0.9.4 released

I have just released the new version 0.9.4 of the xtext-utils unittesting framework. The release became necessary due to a too restrictive version constraint for Google Guice, which made the framework incompatible with the upcoming Xtext 2.3.0 release.

This little framework is a useful addition to Xtext which makes writing DSL tests quite easy. In my daily work on developing Xtext projects and coaching Xtext in workshops it became a useful tool which complements Xtext’s test support. It is not the only way to write tests for Xtext DSLs, but a real simple one. The main idea is that the best way to develop DSLs is to write prototype models. This framework allows you to read, parse, validate, format, serialize, and compare your models in a single line.

The framework can be installed from its update site.

Unofficial release 2.2.2 of the Xtend Maven Plugin

In my previous post I showed how to use the maven-xtend-plugin to compile Xtend sources within a Maven build. This worked fine in simple cases, but there was a limitation of only one supported source directory (the one configured by build/sourceDirectory property), see Bug#367914. Unfortunately already each Xtext project has already 2 source folders (src and src-gen), so this made the plugin unusable for Xtext projects and forced checking in the generated Java files. This causes terrible problems with version control, especially when working on a team.

In the meantime Bug#367914 was resolved, but only for the upcoming Xtend 2.3 (Eclipse Juno) version. This urgent bugfix was not available on a Maven repository. But I needed it for project Spray to finally get rid of the merge conflicts and delete the generated sources from the repository (issue#94). Spray is still using Xtext/Xtend version 2.2.1 and depends on public available Maven artifacts. Since the bugfix won’t be available from the Xtend project for the 2.2.1 release and Eclipse Juno is still some way to go I decided to backport the current 2.3.0 M6 sources of the Xtend Maven Plugin to 2.2.x and create an unofficial version 2.2.2. This plugin I have deployed to the Fornax Repository to make it public available.

For an usage example refer to the POMs of project Spray: pom.xml, parent pom.xml.

JvmType import check

Several Xtext based DSLs will use import statements that import JvmTypes, be them real Java types, or just inferred types produced with your IJvmModelInferrer. The check for a valid import might be simple, but here it is to just make it even easier for you. I just had the need to realize this to resolve Spray issue#113. Basically you need to get the IJvmTypeProvider, to which you gain access through the IJvmTypeProvider.Factory.

public class SprayJavaValidator extends AbstractSprayJavaValidator implements IssueCodes {
    @Inject
    private IJvmTypeProvider.Factory typeProviderFactory;
    @Check
    public void checkImports(final Import imp) {
        // don't check wildcard imports
        if (imp.getImportedNamespace().endsWith(".*"))
            return;
        IJvmTypeProvider typeProvider = typeProviderFactory.findOrCreateTypeProvider(imp.eResource().getResourceSet());
        JvmType jvmType = typeProvider.findTypeByName(imp.getImportedNamespace());
        if (jvmType == null) {
            error("The import " + imp.getImportedNamespace() + " cannot be resolved", SprayPackage.Literals.IMPORT__IMPORTED_NAMESPACE, IMPORT_NOTEXISTS, new String[0]);
        }
    }
    ...
}

That’s all.