Lazy evaluation in Xpand

Usually code generation is a purely sequential process. Since the model does not change during the generation of an artifact all content can be computed in the template where it is needed for the output. But sometimes there is the wish to defer the output to a later point of time during the generation of an artifact.

The typical use case for this is import statements. If for example you want to generate a Java class and want to import all used types then the following alternatives are given:

  • Compute the types that the about-to-be-generated class will use
  • Print out all type names full qualified whenever needed and organize the imports with a postprocessor. For Java code generation the Hybridlabs Beautifier is used widely.

However, both approaches do not seamlessly solve the problem. What really is needed is some kind of lazy evaluation in Xpand. Therefore Jos Warmer wrote a feature proposal once. The feature that he proposed for Xpand is called Insertion Point. The idea was to mark some point in the Xpand template where some code will be inserted at a later point of time. Code is evaluated into this insertion point when the content can be derived easier.

From this feature proposal Feature Request #261607 was created in Eclipse Bugzilla system. In this bug entry, and also offline, a lively discussion arose in the team. The challenges for this feature request were:

  • The Xpand language has a rather small set of keywords. Adding this feature, which is used in some cases only, should not introduce too much changes to the Xpand language
  • An implementation should not break existing code
  • The solution should enable decent evaluation, e.g. avoiding duplicates, sorting

The latest proposal just introduces just one new keyword in Xpand, but requires an implementation pattern with Xtend function. The proposal is to add a keyword ONFILECLOSE to the EXPAND statement. By calling EXPAND with ONFILECLOSE the evaluation of the EXPAND statement is deferred until the FILE statement is closed. Any state that is used by the called definition is computed during the FILE evaluation. The EXPAND statement has to be evaluated with the execution context which is active when reaching the EXPAND statement.

Let’s see this by example. As example we take the project that Xpand’s project wizard creates, with small changes. The entities and types have now an additional ‘packageName’ attribute, and the both entities have been assigned different packages ‘entities1’ and ‘entities2’. Additionally entity ‘Person’ has a feature ‘birthday’ of type Date, which is mapped to java.util.Date. Therefore class has to import entities2.Address and java.util.Date. The used types should be collected when rendering instance variables and accessor methods, but inserted earlier in the code.

First take a look at the template code:

As you can see the definition ImportBlock is called for each Type instance (e.g. Entity and DataType instances) in the collection returned by the Xtend function UsedType(). In an alternative approach this function would be responsible to compute all the types that will be used by this Entity instance. But for the new implementation it just creates an empty list and returns that one. The implementation of the UsedType() function (in file GeneratorExtensions.ext) is:

create List[Type] UsedType (Entity e) : (List[Type]) {};

So at the time the Xpand engine reaches EXPAND ImportBlock… the list would be empty. Now note the ONFILECLOSE keyword at the end of the EXPAND statement. This one tells the engine now that the evaluation of this Xpand statement should be deferred until the file is about to be closed. During evaluation of the template code, in the FOREACH loop, another extension function addUsedType(f.type) is called. This one adds the type of the current processed feature to the collection returned by UsedTypes(). Therfore it is important that the UsedType() function uses the create keyword, since we want to create the collection on first access for one Entity and return the same instance when UsedTypes() is called later for that Entity again.

The function addUsedType() is used in the template like this:


Xpand would print out the result of the function as string, but we don’t want to produce any output by calling the function. Therefore we assure that the function adds the type to the collection and returns an empty string:

addUsedType (Entity e, Type t) : UsedType(e).add(t) -> "";

During evaluation it could be that types were used multiple times within the template, but we want just one import statement per type. Further we don’t need imports for types from the java.lang package (here the packageName information for the DataType instances String and Integer is null) or for types that are in the same package like the entity. Therefore we transform the UsedType() collection before finally invoking the ImportBlock definition.

«EXPAND ImportBlock FOREACH UsedType()
 .select(t|t.packageName!=null && t.packageName!=this.packageName).toSet()
 .sortBy(t|t.qualifiedName()) ONFILECLOSE»


Lazy evaluation allows code generated with Xpand in an non-sequential matter. The proposed solution using ONFILECLOSE solves the desired Insertion Point feature by adding just one additional keyword to the Xpand language. It does not break existing template code. When accepted this code will be contributed to Xpand 0.8.0-M6 soon. For those who want to test it I have created a feature patch for the org.eclipse.xpand feature. The example project with the sources listed in this article can be downloaded here.

Feedback is welcome and is best placed in the bugzilla feature request.


8 thoughts on “Lazy evaluation in Xpand

  1. Pingback: Lazy evaluation feature for Xpand | techscouting through the news

  2. Hello Karsten,

    I was facing a similar problem for C Code generation using Xpand. The variable declarations had to be collected at the top of the method before any other statements could follow. I used ONCLOSEFILE keyword for expanding the variable declarations just as you did for file imports in java. But here, when i run the generator nothing gets out putted to the gen file for that xpand tag. Is there anything else that i have to do for the ONCLOSEFILE keyword to work ?

    • Basically not. I could imagine that the call tree could be something to take into account. Xpand creates execution contexts that are cloned, pushed to stack and popped after execution. The execution context is cloned on evaluating the EXPAND statement. So try to change the call hierarchy.

      If you can reproduce something that does not work, please create a sample project and file a bug.

  3. The ONFILECLOSE keyword can only be used within the FILE tag i guess.
    I had used it in another <><> sequence which is supposed to be called from within the FILE tag. So it was not working. I had refactored code to keep the Xpand with ONFILECLOSE within the FILE tag and it works like a charm 🙂

    Thanks Karsten.

      • Yes. ONFILECLOSE is great for lazy evaluation but consider a scenario where the code generated has to be something like this.
        var declarations;

        method calls and other computations;

        var declarations;

        method calls and other computations;

        and so on… and for repetitive generation of methods i have a FOREACH tag processing a list of model elements within a FILE tag. i would not be able to use the ONFILECLOSE keyword within the loop. Can you please suggest ways of doing this … Would be really grateful if u can give me some ideas.

        Thanks …

  4. Pingback: All Things of World

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s