One Xpand user less, one happy Xtend user more

This week I am consulting a customer who had introduced a model based development approach based almost 10 years ago and used it with success since then. At the time back then Xpand was the most powerful code generation engine and UML models were often used to generate code from. Xtext did not even exist at that time. The customer uses Enterprise Architect and the Enterprise Architect exporter from the components4oaw. The last release was in 2011 and the project has not been developed further. For my customer the component just did what it should, so there was no direct need to change anything at the process. The EA exporter has its flaws, especially since it needs an Enterprise Architect installation and this means it only works on Windows machines. For Enterprise Architect users who do model based development, we therefore offer the YAKINDU Enterprise Architect Bridge, which scales better and can process .eap models on any platform.

My task was to help the customer to modernize their tool chain. To be fair, Xpand is not the right choice anymore. The Xtend Language combines the strengths from Xpand (great templating support, functional programming, static typing, polymorphic dispatch, mature Eclipse tooling) and resolved some weaknesses (performance, compiled code instead of interpreted, Java integration, extensibility of expressions).

For the customer who never had used Xtend so far, but was quite familiar with Xpand, it was quite a surprise how close both languages really are. Most of the concepts can be mapped 1:1 from good old Xpand to Xtend. We created a small generator from scratch and copied functions and templates and translated them manually for demonstration. They understood Xtend within minutes then. Most of the work is monkey-see-monkey-do. For such cases I wrote a small migration script which can translate Xpand,Xtend(1) and Check code to Xtend(2) classes. We used that script now for an initial translation.

One of the reasons why this migration script is not published is that cannot translate Xpand code completely to Xtend. The tool parses Xpand templates and traverses the AST to transform the expressions to Xtend equivalents. But it does not know about the type system which is used by the generator.

And here Xtend is not as powerful as Xpand, especially when using UML. In Xpand, the UML type system adapter analyzed the applied profiles of a model to create virtual types for stereotypes. Elements with stereotypes applied can be processed as if they were of a subtype of the extended type, and tagged values became attributes. In Xtend, there is only the Java type system, and for processing UML models this means that templates have to use the UML metamodel directly. The old “feeling” of a real type system can be simulated with a set of extension functions. I usually introduce an extension class per profile which offers for example a method per tagged value. The creation of such an extension class can again be automated by generating it from a profile .uml model.

Another disadvantage against the Xpand framework is that Xtend is not a code generation framework, but a general purpose programming language. How a generator component looks like that invokes the templates, where to produce output to, how to integrate constraint checks, this all is not provided.

What I usually do here is to use Xtext’s infrastructure like the IGenerator interface and validation based on Xtext’s AbstractDeclarativeValidator and @Check annotations. I reuse Xtext’s MWE Reader and Generator component. However, this requires some work and advanced knowledge. The basic approach is here to make UML models recognized by Xtext as a generic EMF resource. To actually use the Xtext components, a generator specific Guice module has to be created which extends AbstractGenericResourceRuntimeModule and satisfy several dependencies which an Xtext language already as configured by default. A good part of the approach was described by Christian Dietrich in his blog posts “Xtend2 Code Generators with Non-Xtext Models” and “Xtext 2.0 and UML“. I may go into details in a later blog post.

Although Xtend is such a natural choice for writing code generators and it is so well-integrated in the Xtext ecosystem, there is framework support missing for non-Xtext models. It is easy to write a trivial framework, but why should everyone start writing their own when good infrastructure already exists in the Xtext framework? From what I experienced again, there is need to have some more framework support for this use case. I doubt that it could be part of Xtext itself, but maybe we will provide a framework for Xtend based code generators in the future.

At the end we were able to show and demonstrate a migration path in one single workshop day. The customer was happy to save several days or weeks of time. The workshop costs were compensated by far for them. From a sales perspective it might not be wise to leave a customer in a state where he is not dependent on our services in the next time, but this is not how we work. For me it feels right.

Give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime.

Fornax Workflow Plugin 3.4.0 released

Finally I have managed to release a new version of the Fornax Maven Workflow Plugin again. This Maven plugin allows execution of MWE and MWE2 workflows from Maven, which is typically the case if you want to automate the build of Xtext or Xpand based projects with Maven.

The new version has some interesting new features which may be worth upgrading:

  • m2e lifecycle metadata
  • workflow dependencies can be configured as plugin dependencies
  • more options to detect changes that trigger the build
  • force/skip parameter to configure workflow execution overrides per configuration
  • log level detection when running in forked JVM

I have especially to thank Dieter Schulten from escalon, who has been willing to contribute to this release. He has mainly contributed the m2e configuration and some tests for the plugin.

Xpand builder deadlock situation

Today a colleague in my project came with an issue she experienced with several workspaces, which contain amongst others Xpand projects. After starting Eclipse during the initial build phase Eclipse hang and did nothing. In the workspace log we found indication for a deadlock situation, since there one of the last entries stated that thread Main was waiting for thread Main, and this happened during class loading of the XtendXpandBuilder. We debugged the situation and from the stacktraces of the running threads we could see that one thread tried to refresh a resource that was out of sync.

The problem was solved by starting Eclipse with the -refresh platform option. The out-of-sync resource was synchronized before the builder runs.

Prototyping a Configuration Toolkit for Eclipse SMILA with Xtext

I got the chance to get a bit more familiar with Eclipse SMILA and started development of a configuration toolkit with Xtext. Target is to develop a prototype which enables an easier setup of a valid SMILA configuration by use of a textual DSL with all the benefits which you get from using such a DSL, like semantic validation, content assist etc. SMILA is configured by a bunch of XML files conforming to defined XSDs. Sometimes information is spread around different configuration files, and misconfiguration leads to runtime errors or even to no error at all.

But lets start with SMILA first…

What is SMILA about?

SMILA is an extensible framework for building search solutions to access unstructured information in the enterprise. Besides providing essential infrastructure components and services, SMILA also delivers ready-to-use add-on components, like connectors to most relevant data sources. Using the framework as their basis will enable developers to concentrate on the creation of higher value solutions, like semantic driven applications etc.

To give a rough imagination: You can configure different kinds of agents which search media for information (e.g. files, web pages etc.), and relevant data is extracted from those resources and published to some queue (ActiveMQ is used by default). Listeners react on entries and execute BPEL processes to process the information. Final goal is to index the data in stores, which can be searched by clients. Lucene is used by default as indexing engine.

Getting SMILA running

The SMILA project provides distributions for Windows and Linux. Since I’m working on a Mac I could not use them. So I followed the development guideline to setup a dev environment. In my fresh workspace I checked out first the trunk, but switched back to tag 0.5-M3 to have the same state as the distributions.

After finishing the checkout I finally was able to follow again the good 5 Minutes to Success tutorial. But don’t expect you can finish the tutorial in 5 minutes ūüėČ One word to mention: SMILA requires Java 6, and my development IDE is started by default with Java 5. So I needed to configure Java 6 for my target platform and also had to add the RCP delta pack, since 1.6 requires 64 bit libraries on Mac.

Contained in the sources is a example configuration project SMILA.application, which can be started by a launch configuration in the SMILA.launch project. Here is a small screenshot of the SMILA.application project structure.

The application contains several XML configuration files and their XSDs in a structure which reflects the plugins that are used. The tutorial explains small changes to the configuration and which files have to be changed, but for setting up a brand new project it might become more complicated if one is not familiar with the structure.

Starting the prototype

First I have to make clear that the following is early development state. I plan to extend the functionality when getting some time again. Since I’m involved often at customers I cannot tell how fast I progress now. At least I get the possibility to spend some days in the near future on it, so I’m expecting to have something useful in the near future.

I created the Xtext projects for the SMILA DSL and added some first rules. After running the MWE workflow Xtext generated the project infrastructure.

SMILA project wizard

When looking at the example project I recognized that a normal project setup would require copy/paste of an existing one and changing some files. Therefore extending the generated project wizard seemed to be a good starting point. The extended wizard now lets you set up a SMILA application with all the required files.

After finishing the wizard a project in the workspace is created. All static resources (esp. project structure and XSDs) are copied from the UI plugin into the new project and as a start some files are generated using Xpand with the information filled in into the wizard.

The wizard generated from the SimpleProjectWizardFragment was not so extensible for my case as it should be, so I had to copy some code from the generated classes and provide a manual implementation with some copied code. I think the fragment could be changed easily to improve and I will set up a change request on that later and post it to bugzilla.

At the moment the project wizard generates the following artifacts from the information provided on the pages:

  • SMILA DSL model file
  • log4j.properties
  • Launch configuration
  • Tomcat server config

Here you can see the project the wizard created:

Crawler configuration

The first configuration I targeted to describe with the DSL is the configuration of the FileSystemCrawler and FeedAgent. This is pretty straight forward, nearly 1:1 mapping. Here’s an excerpt from the appropriate configuration file “feed.xml” shipped with the example:

<DataSourceConnectionConfig
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:noNamespaceSchemaLocation="../org.eclipse.smila.connectivity.framework.crawler.filesystem/schemas/FileSystemDataSourceConnectionConfigSchema.xsd"
>
  <DataSourceID>file</DataSourceID>
  <SchemaID>org.eclipse.smila.connectivity.framework.crawler.filesystem</SchemaID>
  <DataConnectionID>
    <Crawler>FileSystemCrawler</Crawler>
  </DataConnectionID>
  ...
  <Attributes>
    <Attribute Type="Date" Name="LastModifiedDate" HashAttribute="true">
      <FileAttributes>LastModifiedDate</FileAttributes>
    </Attribute>
    <Attribute Type="String" Name="Filename">
      <FileAttributes>Name</FileAttributes>
    </Attribute>
    <Attribute Type="String" Name="Path" KeyAttribute="true">
      <FileAttributes>Path</FileAttributes>
    </Attribute>
    <Attribute Type="String" Name="Content" Attachment="true">
      <FileAttributes>Content</FileAttributes>
    </Attribute>
    <Attribute Type="String" Name="Extension">
      <FileAttributes>FileExtension</FileAttributes>
    </Attribute>
    <Attribute Type="String" Name="Size">
      <FileAttributes>Size</FileAttributes>
    </Attribute>
  </Attributes>
  <Process>
    <BaseDir>/Users/thoms/temp</BaseDir>
    <Filter Recursive="true" CaseSensitive="false">
      <Include Name="*.txt"/>
      <Include Name="*.htm"/>
      <Include Name="*.html"/>
      <Include Name="*.xml"/>
    </Filter>
  </Process>
</DataSourceConnectionConfig>

And here the same situation described in the DSL (the box with “caseSensitive” is there because I pressed CTRL+SPACE after the keyword “recursive” and the content assist proposes that “caseSensitive” could be entered here):

I decided that the record attribute name (in XML the Attribute#name property) can be omitted in the case that it matches the File attribute name, which I think often will be the case. Only if the names don’t match a mapping has to be done. Here the example is

FileExtension -> Extension

“FileExtension” is the File attribute name and “Extension” is the name of the Record attribute.

Flags are added in brackets and are optional (key, hash, attachment).

Builder Integration

Since Xtext Helios M4 a builder infrastructure was added to Xtext. I leveraged this infrastructure to generate the resulting configuration files on-the-fly when you save the DSL model. So if you, for example, add an “Include” line to your model the respective crawler config is automatically changed. Even better: When you rename your crawler, let’s say from “file” to “userdir_scanner” the configuration file “file.xml” gets deleted from your workspace and is replaced by “userdir_scanner.xml”!

After renaming the FileSystemCrawler:

Outlook

This is just the start of this project and many things have to be done now. I plan to use this project also as a good example for using the Xtext features properly, of course open sourced. Also I have to learn more about SMILA and the appropriate configuration. I’m in exchange with Sebastian Voigt from brox, co-lead of the SMILA project. With his help I think this project can be a valuable contribution to SMILA later.

Here are some features that I want to add to this project:

  • Complete language for covering the tutorial
    In a first step at least everything that makes up the “5 Minutes to Success” tutorial should be possible to describe in the DSL and the configuration files should be generated from that description.
  • Integrate existing configuration files
    I saw that some configuration files might not be worth to be mapped to a DSL and might be better left just in XML for editing. One example is the QueueWorkerConnectionConfig.xml, where the available brokers are defined. Of course from the DSL I want to refer to brokers at several places and I need to get them from this file. My first idea is here to use generate EMF models from the XSDs using the XSD importer. That makes it possible to reference types from that schema directly in the DSL. It should be like normal integration of existing Ecore models.
  • Validation
    One of the major benefits that the DSL can provide is the ease to add validation on the models. Especially consistency constraints make sense, for example to prove that every queue where records are routed to must have a listener that processes that records further.
  • JDT integration
    In the BPEL configuration files services are invoked. The services are qualified by their class names and parameters that can be passed correspond to properties. At Eclipse Summit 2009 Sven Efftinge and Sebastian Zarnekow showed a nice integration of Xtext with JDT to add content assist and validation on qualified Java classes.
  • Product build
    The complete bundle, SMILA and the Configuration toolkit, should be available as a ready-to-use product. I’m planning to use Maven Tycho for setting up the build process.

These are just some few examples of what I can imagine for the future. I hope that I find or get some time to realize this.