Xtend – Generating from multiple input models

Xtext’s integration of Xtend based code generators is nice. If you save changed (or add/remove) DSL model files an incremental build will automatically be triggered and even removes derived artifacts if you remove the source from which it is generated. The IGenerator interface takes a Resource, and you can generate 1 or multiple target artifacts from the model elements within that resource.

However, the API does not allow to create a single artifact from multiple input models. But this is a common use case if you do serious code generation. I will show you how you could extend Xtext’s Domainmodel example to generate a simple text file that lists the name of all Entity instances of all .dmodel files that are reachable in the built project.

Adding an extended generator interface

Create an interface IGenerator2 that extends the IGenerator interface:

public interface IGenerator2 extends IGenerator {
	/**
	 * @param input - the input for which to generate resources
	 * @param fsa - file system access to be used to generate files
	 */
	public void doGenerate(ResourceSet input, IFileSystemAccess fsa);
}

The generator class will later implement this interface and generate a single artifact from all Entity instances in all resources.

Extending the JavaProjectBasedBuilderParticipant

The JavaProjectBasedBuilderParticipant is used by default to invoke the IGenerator implementation that was bound to the runtime module. We will extend this class to add additional logic and inject an IGenerator2 instance that will be invoked once per triggered build.

The builder participant is an UI concept, so create the extended class JavaProjectBasedBuilderParticipant2 in the UI plugin:

package org.eclipse.xtext.builder;

import java.util.List;

import org.eclipse.core.runtime.CoreException;
import org.eclipse.core.runtime.IProgressMonitor;
import org.eclipse.emf.ecore.resource.Resource;
import org.eclipse.xtext.builder.JavaProjectBasedBuilderParticipant;
import org.eclipse.xtext.generator.IFileSystemAccess;
import org.eclipse.xtext.generator.IGenerator2;
import org.eclipse.xtext.resource.IContainer;
import org.eclipse.xtext.resource.IResourceDescription;
import org.eclipse.xtext.resource.IResourceDescription.Delta;
import org.eclipse.xtext.resource.IResourceDescriptions;
import org.eclipse.xtext.resource.impl.ResourceDescriptionsProvider;

import com.google.inject.Inject;

public class JavaProjectBasedBuilderParticipant2 extends JavaProjectBasedBuilderParticipant {
  @Inject
  private ResourceDescriptionsProvider resourceDescriptionsProvider;

  @Inject
  private IContainer.Manager containerManager;

  @Inject (optional=true)
  private IGenerator2 generator;

  protected ThreadLocal<Boolean> buildSemaphor = new ThreadLocal<Boolean>();

  @Override
  public void build(IBuildContext context, IProgressMonitor monitor) throws CoreException {
    buildSemaphor.set(false);
    super.build(context, monitor);
  }

  @Override
  protected void handleChangedContents(Delta delta, IBuildContext context, IFileSystemAccess fileSystemAccess) {
    super.handleChangedContents(delta, context, fileSystemAccess);
    if (!buildSemaphor.get() && generator != null) {
      invokeGenerator(delta, context, fileSystemAccess);
    }
  }

  @Override
  protected void handleDeletion(Delta delta, IBuildContext context, IFileSystemAccess fileSystemAccess) {
    super.handleDeletion(delta, context, fileSystemAccess);
    if (!buildSemaphor.get() && generator != null) {
      invokeGenerator(delta, context, fileSystemAccess);
    }
  }

  private void invokeGenerator (Delta delta, IBuildContext context, IFileSystemAccess fileSystemAccess) {
    buildSemaphor.set(true);
    Resource resource = context.getResourceSet().getResource(delta.getUri(), true);
    if (shouldGenerate(resource, context)) {
      IResourceDescriptions index = resourceDescriptionsProvider.createResourceDescriptions();
      IResourceDescription resDesc = index.getResourceDescription(resource.getURI());
      List<IContainer> visibleContainers = containerManager.getVisibleContainers(resDesc, index);
      for (IContainer c : visibleContainers) {
        for (IResourceDescription rd : c.getResourceDescriptions()) {
          context.getResourceSet().getResource(rd.getURI(), true);
        }
      }

      generator.doGenerate(context.getResourceSet(), fileSystemAccess);
    }
  }
}

Note that IGenerator instance will be invoked once per changed resource, so potentially multiple times. We have to make sure that the IGenerator2 instance will be invoked just once. Therefore a thread local semaphor flag will be queried before the generator is called. In this scenario it does not matter whether a change or deletion should be handled. The generator always processes the whole model once.

The most important thing is to populate the BuildContext’s ResourceSet with all relevant model resources. It is important here collect the resources that are in scope of the built project. The right way to do this is to use the language’s IContainer.Manager to collect the resources that are within the visible containers. Each of them is queried for its contained IResourceDescription instances, from which the Resource instances are created.

Extending the generator class

Now open the generator class DomainmodelGenerator and implement IGenerator2. Add an implementation of the method from the extended interface:

...
// makes allContentsIterable available
import static extension org.eclipse.xtext.xtend2.lib.ResourceExtensions.*
...

class DomainmodelGenerator implements IGenerator2 {
  ...
  override void doGenerate(ResourceSet rs, IFileSystemAccess fsa) {
    val Iterable<Entity> entities = rs.resources.map(r|r.allContentsIterable.filter(typeof(Entity))).flatten

    fsa.generateFile("entities.txt", entities.compile)
  }

  def compile (Iterable<Entity> entities) '''
    «FOR e : entities SEPARATOR ", "»«e.name»«ENDFOR»
  '''

  ...
}

Guice configuration

Now bind the extended builder particpant in the UI module:

public class DomainmodelUiModule extends AbstractDomainmodelUiModule {
	@Override
	public Class<? extends IXtextBuilderParticipant> bindIXtextBuilderParticipant() {
		return JavaProjectBasedBuilderParticipant2.class;
	}
}

… and the generator to the Runtime module:

public class DomainmodelRuntimeModule extends AbstractDomainmodelRuntimeModule {
	public Class<? extends IGenerator2> bindIGenerator2 () {
		return DomainmodelGenerator.class;
	}
}

Result & Conclusion

After restarting your runtime workspace the generator will produce an “entities.txt” file to the src-gen folder of a project with .dmodel files, which will simply list the names of all found Entity instances.

This scenario is quite common, so it would be great if the Xtext framework could provide built-in support for that. I have therefore raised a feature request.

Advertisements

17 thoughts on “Xtend – Generating from multiple input models

  1. Pingback: Xtend – Generating from multiple input models | Eclipse | Syngu

  2. I have two remarks:

    The import:
    import static extension org.eclipse.xtext.xtend2.lib.ResourceExtensions.*
    is essential if you want to use allContentsIterable.
    For me as a new Xtext/xtend user I had to figure out why allContentsIterable was not available on my Resource.

    Using Eclipse 3.7R4 with Xtend 1.1.0.v201106060417 org.eclipse.xtend.feature.group
    I have to add the

    public Class bindIGenerator2 () {
    return DomainmodelGenerator.class;
    }

    to DomainRuntimeModule, not in the UI DomainmodelUiModule.

    Thank you for the documentation, it works!

  3. Pingback: Xtend - NUTRITION STORES – NUTRITION STORES

  4. Hi Karsten,

    really interesting article.

    Your approach works only with “automatically” UI triggering on save, right?

    What would be a correct solution when triggering the generation process via workflow?

    My approach is to use a subclass of org.eclipse.xtext.generator.GeneratorComponent. In my generator class I collect the resources (on everey call of doGenerate(Resource resource, IFileSystemAccess fsa)).

    At the end of GeneratorComponent I call a method on the the generator via overriding “postInvoke()”

    However, this seems rather unclean…

    TIA
    Martin

    • Hi Martin,

      right, the article explains only the UI side. In standalone execution a custom workflow component is the right approach. I would use the same extended interface and prove, if the IGenerator is an instance of the extended interface. In that case, call the doGenerate(ResourceSet, IFileSystemAccess) method. The ResourceSet could be retrieved from any Resource, or by creating an own ResourceSet with the Resources from the model slot. The postInvoke() method would be the right place.

      I don’t consider this to be unclean, since there is no builtin way to achieve it. The given approach does work and does not violate any API.

      Best,
      ~Karsten

  5. Hi Karsten,

    thanks for sharing this – your solution was very helpful for us.

    I just have two issues:

    1. I my scenario I run sometimes in the exception “java.net.MalformedURLException: unknown protocol: java” when invokeGenerator is called. I debugged it and found out that the uri of delta is java:/Objects/de.test.model.impl.Entity1000Imp. When trying to get the resource in line 55 of your example the problem occurs. I fixed it by checking if the uri of delta is a platform resource:
    if (delta.getUri().isPlatformResource()) {
    buildSemaphor.set( true );

    }
    I wonder why one of my generated Java-classes gets into the XtextBuild. Can you image why this might happen?

    2. I use Xtext 2.1 and JavaProjectBasedBuilderParticipant is marked deprecated. I replaced it with BuilderParticipant and changed the handleChangedContents-method:

    @Override
    protected void handleChangedContents( IResourceDescription.Delta delta, IXtextBuilderParticipant.IBuildContext context,
    EclipseResourceFileSystemAccess2 fileSystemAccess ) throws org.eclipse.core.runtime.CoreException {
    super.handleChangedContents( delta, context, fileSystemAccess );
    if (!buildSemaphor.get() && generator != null) {
    invokeGenerator( delta, context, fileSystemAccess );
    }
    }

    and it works fine. But BuilderParticipant has no handleDeletion-method and the build is not triggered on deletion of a file. Any hints how I can fix this problem are highly appreciated.

    Cheers!
    Annette

    • Hi Annette,

      1) I’m not completely sure here. Might be because of JVM model inference.
      2) You could register a IResourceChangeListener when starting the UI Activator.

      Kind regards,
      ~Karsten

      • Karsten, when I downloaded the source from github, the manifest.mf has errors. There are references to org.eclipse.xtext.example.domainmodel.domainmodel (note two segments named domainmodel), but the source on github only has packages with a single domainmodel segment. Also, the plugin.xml references the following, which does not exist: org.eclipse.xtext.example.domainmodel.domainmodel.DomainmodelPackage.

        Are there projects missing from what you placed on github, or am I misunderstanding something?

        Thanks!

      • The problem seems to be that the GenerateDomainmodelLanguage.mwe2 needs to be run. But, when I do so, there was an error that is resolved by adding this line to the StandaloneSetup section:

        registerGeneratedEPackage = “org.eclipse.xtext.xbase.XbasePackage”.M19mwyltm77!

        There are other problems, but that is fine for now.

  6. I have solved this problem by using the Xtext Index to find all elements of the type Entity. The advantage is that you do not have to open and parse all Xtext resources and thus this should be much more efficient. The disadvantage is that this does not work from the command line outside Eclipse, because I have not found a way to access an Xtext Index outside Eclipse.

  7. Thx you very much for your hard work. It helped me a lot but could u try to use better names for ur variables 😀 (IGenerator and IGenerator2 ….)

    • You will find this naming scheme all over Eclipse. Eclipse projects have some restrictions in API compatibility, and often interfaces are not changed, but new functionality is provided by subclassed interfaced. There the simple naming scheme is often used to add a counter to the extended interfaces’ name. In Eclipse, search for types with name pattern “I*2” and you’ll wonder how often you’ll find it.

  8. Pingback: Xtext Generation from multiple model files | ASK Dev Archives

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s