Xtext 2.1: using Xbase expressions

On a previous post, I started to blog about how Xtext 2.1 made it extremely easy to have a powerful integration with Java in your DSL. In this post I’d like to continue to experiment with this mechanism, and in particular I’ll start to use Xbase, a new expression language library which allows to integrate (Java-like) expressions in your DSL.

If you take a look at the Five simple steps to your JVM language, you’ll see the power of Xbase! Moreover, in Xtext 2.1, using Xbase is even easier since by writing only a AbstractModelInferrer you’ll get a full integration with Java (by mapping your AST model elements to Java concepts) and a generator (which will produce Java code corresponding to this mapping). Thus, you won’t even need a generator at all.

However, in this post, I’d like to inspect how to use only a small part of Xbase and still have the control on the generation part: in particular (for other projects) I would like to retain the control on the generation for my model, while relying on the Xbase generation for the Xbase parts. Thus, in this post I’ll describe:

• how to integrate Xbase expressions (XExpression) in your DSL
• write a generator for your DSL and reuse the XbaseCompiler for the code of XExpressions

Following the spirit of the previous post, I won’t use the Domainmodel example, but something really simple, like the Greeting example, i.e., the very basic DSL you will get when creating an Xtext project inside Eclipse. In particular, you might want to go through the previous post before reading this post (since we reuse some of the concepts seen there).

So, first of all, create a new Xtext project; I’ll call this project org.xtext.example.helloxbase (and the files will have extension helloxbase). Now, we can start generate Xtext artifacts. With the defaults, you’ll also get a HelloXbaseGenerator.xtend, which we now to generate Java code. The first implementation of the generator is: for each Greeting element we will obtain a package and a class named according to the Greeting name feature (package all lower case, and class name with the first letter capital). Moreover, the generated main method will print the corresponding greeting:

package org.xtext.example.helloxbase.generator

import org.eclipse.emf.ecore.resource.Resource
import org.eclipse.xtext.generator.IFileSystemAccess
import org.eclipse.xtext.generator.IGenerator
import org.xtext.example.helloxbase.helloXbase.Greeting

import static extension org.eclipse.xtext.xtend2.lib.ResourceExtensions.*

class HelloXbaseGenerator implements IGenerator {

	override void doGenerate(Resource resource, IFileSystemAccess fsa) {
		for(greeting: resource.allContentsIterable.filter(typeof(Greeting))) {
			fsa.generateFile(
				greeting.packageName + "/" + // package
				greeting.className + ".java", // class name
				greeting.compile)
		}
	}

	def compile(Greeting greeting) '''
	package «greeting.packageName»;

	public class «greeting.className» {
		public static void main(String args[]) {
			System.out.println("Hello «greeting.name»");
		}
	}
	'''

	def className(Greeting greeting) {
		greeting.name.toFirstUpper
	}

	def packageName(Greeting greeting) {
		greeting.name.toLowerCase
	}
}

If you’re already familiar with Xtend2 (or if you read the previous post), you should have a clue about what the generator does; letÔÇÖs try the generator: restart your second eclipse instance, and make sure that in the plugin project you had created with the helloxbase file, you have a folder named src-gen, and that folder is configured as a source folder. If you type something like

Hello foo!
Hello bar!

you should see some Java code generated in the src-gen folder (according to the schema detailed above).

Now, the idea is to enrich the HelloXbase greeting DSL with some Xbase expressions, thus, we modify the grammar of the language (HelloXbase.xtext) as follows (you should be familiar with ‘import’ functionalities of Xtext):

grammar org.xtext.example.helloxbase.HelloXbase with 
	org.eclipse.xtext.xbase.Xbase

generate helloXbase "http://www.xtext.org/example/helloxbase/HelloXbase"

Model:
	imports += Import*
	greetings+=Greeting*;

Import:
  'import' importedNamespace = QualifiedNameWithWildcard
;

QualifiedNameWithWildcard:
  QualifiedName '.*'?
; 

Greeting:
    'Hello' name=ID 'from' expression = XExpression '!'
;

Note that we now derive our grammar from the Xbase grammar, so we can now rely on Xbase grammar rules; the idea is to be able to write in our DSL something like

Try it yourself: regenerate the Xtext artefacts, and restart the runtime eclipse instance. To fully enjoy Xbase syntax, please make sure that in the project you created in the runtime workspace you also have org.eclipse.xtext.xbase.lib as a dependency in the MANIFEST.

You can see lots of cool stuff here! You basically have all the powerful Xbase expression Java-like syntax, access to static methods using ::, closures, and extension methods (the nullOrEmpty from StringExtensions)! All for free!

Now, what happened to our generator? Is it still invoked automatically on resource changes? I’m afraid not: as soon as you derived your grammar from Xbase, a new file appeared in your project, HelloXbaseJvmModelInferrer.xtend, since in Xtext 2.1, as hinted at the beginning, the generation strategy changed. In particular, in the src-gen AbstractHelloXbaseRuntimeModule the IGenerator is now bound to org.eclipse.xtext.xbase.compiler.JvmModelGenerator; since we want our own generator back, we simply override this binding in HelloXbaseRuntimeModule (the one we’re allowed to modify):

public class HelloXbaseRuntimeModule extends org.xtext.example.helloxbase.AbstractHelloXbaseRuntimeModule {

	/**
	 * Avoids to use the default org.eclipse.xtext.xbase.compiler.JvmModelGenerator
	 * when using xbase.
	 * @see org.xtext.example.helloxbase.AbstractHelloXbaseRuntimeModule#bindIGenerator()
	 */
	@Override
	public Class<? extends IGenerator> bindIGenerator() {
		return HelloXbaseGenerator.class;
	}

}

If you restart your runtime instance you’ll see our generator is back!

Now, we want to generate the Java code corresponding to the Xbase XExpression; we then simply reuse the XbaseCompiler in our generator (which, in Xtext, you know, means ‘have it injected’). In the generator we will now use also the ImportManager (which is explained in the previous post) and start to do some experiments with the XbaseCompiler:

package org.xtext.example.helloxbase.generator

import com.google.inject.Inject
import org.eclipse.emf.ecore.resource.Resource
import org.eclipse.xtext.common.types.TypesFactory
import org.eclipse.xtext.generator.IFileSystemAccess
import org.eclipse.xtext.generator.IGenerator
import org.eclipse.xtext.xbase.XExpression
import org.eclipse.xtext.xbase.compiler.ImportManager
import org.eclipse.xtext.xbase.compiler.StringBuilderBasedAppendable
import org.eclipse.xtext.xbase.compiler.XbaseCompiler
import org.xtext.example.helloxbase.helloXbase.Greeting

import static extension org.eclipse.xtext.xtend2.lib.ResourceExtensions.*

class HelloXbaseGenerator implements IGenerator {

	@Inject
	protected XbaseCompiler xbaseCompiler

	override void doGenerate(Resource resource, IFileSystemAccess fsa) {
		for(greeting: resource.allContentsIterable.filter(typeof(Greeting))) {
			fsa.generateFile(
				greeting.packageName + "/" + // package
				greeting.className + ".java", // class name
				greeting.compile)
		}
	}

	def getJvmType(Greeting greeting) {
		val declaredType = TypesFactory::eINSTANCE.createJvmGenericType
		declaredType.setSimpleName(greeting.className)
		declaredType.setPackageName(greeting.packageName)
		declaredType
	}

	def compile(Greeting greeting) '''
	«val importManager = new ImportManager(true, getJvmType(greeting))»
	«val mainMethod = compile(greeting, importManager)»
	package «greeting.packageName»;
	«IF !importManager.imports.empty»

	«FOR i : importManager.imports»
		import «i»;
	«ENDFOR»
	«ENDIF»

	«mainMethod»
	'''

	def compile(Greeting greeting, ImportManager importManager) '''
	public class «greeting.className» {
		public static void main(String args[]) {
			«compile(greeting.expression, importManager)»

			System.out.println("Hello «greeting.name» from ");
		}
	}
	'''

	def compile(XExpression xExpression, ImportManager importManager) {
		val result = new StringBuilderBasedAppendable(importManager)
		xbaseCompiler.toJavaStatement(xExpression, result, true)
		result
	}

	def className(Greeting greeting) {
		greeting.name.toFirstUpper
	}

	def packageName(Greeting greeting) {
		greeting.name.toLowerCase
	}
}

Let’s concentrate on these parts:

	def compile(Greeting greeting, ImportManager importManager) '''
	public class «greeting.className» {
		public static void main(String args[]) {
			«compile(greeting.expression, importManager)»

			System.out.println("Hello «greeting.name» from ");
		}
	}
	'''

	def compile(XExpression xExpression, ImportManager importManager) {
		val result = new StringBuilderBasedAppendable(importManager)
		xbaseCompiler.toJavaStatement(xExpression, result, true)
		result
	}

We tried to use the toJavaStatement of XbaseCompiler, which seems a good candidate for what we want; in particular, by passing true as the last argument we say that the “result” of generation should be assigned to a Java variable, so that we can refer to it; let’s see what happens in the runtime workspace with this generation:

Hey, cool! The XbaseCompiler generates all the Java statements corresponding to the original XExpression (including closures)! Now we would only like to access the last variable generated (in the examples above, _isNullOrEmpty and _apply, respectively), so that we can generate some Java code ourselves using that variable; but how can we know its name?

Well, the StringBuilderBasedAppendable class not only contains the result of the generation; it also contains a map with all the intermediate Java local variables generated by the compiler. Each variable, in the map, has as the key the corresponding Xbase object. So we only need to know the variable name corresponding to our XExpression and we’re done:

result.getName(greeting.expression)

So the final generator looks like this:

package org.xtext.example.helloxbase.generator

import com.google.inject.Inject
import org.eclipse.emf.ecore.resource.Resource
import org.eclipse.xtext.common.types.TypesFactory
import org.eclipse.xtext.generator.IFileSystemAccess
import org.eclipse.xtext.generator.IGenerator
import org.eclipse.xtext.xbase.XExpression
import org.eclipse.xtext.xbase.compiler.ImportManager
import org.eclipse.xtext.xbase.compiler.StringBuilderBasedAppendable
import org.eclipse.xtext.xbase.compiler.XbaseCompiler
import org.xtext.example.helloxbase.helloXbase.Greeting

import static extension org.eclipse.xtext.xtend2.lib.ResourceExtensions.*

class HelloXbaseGenerator implements IGenerator {

	@Inject
	protected XbaseCompiler xbaseCompiler

	override void doGenerate(Resource resource, IFileSystemAccess fsa) {
		for(greeting: resource.allContentsIterable.filter(typeof(Greeting))) {
			fsa.generateFile(
				greeting.packageName + "/" + // package
				greeting.className + ".java", // class name
				greeting.compile)
		}
	}

	def getJvmType(Greeting greeting) {
		val declaredType = TypesFactory::eINSTANCE.createJvmGenericType
		declaredType.setSimpleName(greeting.className)
		declaredType.setPackageName(greeting.packageName)
		declaredType
	}

	def compile(Greeting greeting) '''
	«val importManager = new ImportManager(true, getJvmType(greeting))»
	«val mainMethod = compile(greeting, importManager)»
	package «greeting.packageName»;
	«IF !importManager.imports.empty»

	«FOR i : importManager.imports»
		import «i»;
	«ENDFOR»
	«ENDIF»

	«mainMethod»
	'''

	def compile(Greeting greeting, ImportManager importManager) '''
	«val result = compile(greeting.expression, importManager)»
	public class «greeting.className» {
		public static void main(String args[]) {
			«result»

			Object expression = «result.getName(greeting.expression)»;
			System.out.println("Hello «greeting.name» from " +
				expression.toString());
		}
	}
	'''

	def compile(XExpression xExpression, ImportManager importManager) {
		val result = new StringBuilderBasedAppendable(importManager)
		xbaseCompiler.toJavaStatement(xExpression, result, true)
		result
	}

	def className(Greeting greeting) {
		greeting.name.toFirstUpper
	}

	def packageName(Greeting greeting) {
		greeting.name.toLowerCase
	}
}

And in the runtime instance you’ll get what you wanted:

You can find the sources for the project helloxbase at

https://github.com/LorenzoBettini/Xtext2-experiments

Hope you find this post useful, and stay tuned for new posts about Xtext 🙂