Xtext 2.1: using Xbase variables

On a previous post, I started to blog about how Xtext 2.1 made it extremely easy to have a powerful integration with Java thanks to Xbase, a new expression language library which allows to integrate (Java-like) expressions in your DSL. I had also blogged about using JVM types from your DSL.

In this post, IÔÇÖd like to continue to inspect how to use only a small part of Xbase and still have the control on the generation part: in particular (for other projects) I would like to retain the control on the generation for my model, while relying on the Xbase generation for the Xbase parts. Thus, in this post IÔÇÖll describe:

• how to integrate Xbase variables (XVariableDeclaration) and expressions (XExpression) in your DSL
• how to extend Xbase scoping for making the variables visible in the expressions
• write a generator for your DSL and reuse the XbaseCompiler for the code of XVariableDeclaration and XExpressions

Following the spirit of the previous post, I wonÔÇÖt use the Domainmodel example, but something really simple, like the Greeting example, i.e., the very basic DSL you will get when creating an Xtext project inside Eclipse. In particular, you might want to go through the previous posts before reading this post (since we reuse some of the concepts seen there):

• the one about JVM types (concerning how to use the ImportManager for generating correct code for accessing Java types and the corresponding import statements)
• and the one about Xbase expressions (concerning how to use the XbaseCompiler for generating code corresponding to Xbase expressions).

So, first of all, create a new Xtext project; IÔÇÖll call this project org.xtext.example.helloxvars (and the files will have extension helloxvars). Now, we can start generate Xtext artifacts. With the defaults, youÔÇÖll also get a HelloXvarsGenerator.xtend, which we now to generate Java code. The basic idea of the generator is: for each Greeting element we will obtain a package and a class named according to the Greeting name feature (package all lower case, and class name with the first letter capital). We’ll deal with the generator later in this post.

Now, the idea is to enrich the greeting DSL with some Xbase variables (before the greeting statements) and some Xbase expressions (used within the greeting statements) which can access the declared variables; thus, we modify the grammar of the language (HelloXvars.xtext) as follows (you should be familiar with ÔÇÿimportÔÇÖ functionalities of Xtext):

grammar org.xtext.example.helloxvars.HelloXvars with 
	org.eclipse.xtext.xbase.Xbase

generate helloXvars "http://www.xtext.org/example/helloxvars/HelloXvars"

Model:
	imports += Import*
	varDeclarations += XVariableDeclaration*
	greetings+=Greeting*;

Import:
	'import' importedNamespace = QualifiedNameWithWildcard
;

QualifiedNameWithWildcard:
	QualifiedName '.*'?
; 

Greeting:
	'Hello' name=ID 'from' expression = XExpression '!'
;

The idea is to end up (at the end of this post) with a DSL which allows us to write programs as follows:

Note that Xbase (its validator) already checks that the types are correct, i.e., that the initialization expression has a type which is a subtype of the declared type for the variable. But, more important, you will reuse the type inference mechanism of Xbase: you don’t need to declare the type for the variable when this can be inferred by Xbase itself!

Regenerate the Xtext artefacts, and start a new runtime eclipse instance. Create a new plugin project (say ÔÇÿhelloxvarsÔÇÖ), and in the source folder we create a .helloxvars file (say ÔÇÿMy.helloxvarsÔÇÖ) and accept to add the Xtext nature. To fully enjoy Xbase syntax, please make sure that in the project you created in the runtime workspace you also have org.eclipse.xtext.xbase.lib as a dependency in the MANIFEST.

Now, if you try to write code like in the above image you’ll get errors due to the Xbase expressions that cannot access Xbase variables; in fact, Xbase variables are automatically visible to Xbase expressions only when they are both part of an XBlockExpression, which we don’t use in our case. Thus, we must extend the Xbase scoping mechanism in order to make all the variables declared before the greetings to all the expressions of the greetings.

As soon as we start to use Xbase in our grammar, the generated runtime module will bind the IScopeProvider to XbaseScopeProvider; so, first of all, we must override this binding in our runtime module, in order to bind the scope provider to the one of our DSL:

public class HelloXvarsRuntimeModule extends
		org.xtext.example.helloxvars.AbstractHelloXvarsRuntimeModule {

	@Override
	public Class<? extends IScopeProvider> bindIScopeProvider() {
		return HelloXvarsScopeProvider.class;
	}

}

And now, we make our scope provider extend the XbaseScopeProvider; by inspecting the code of XbaseScopeProvider we find a possible good candidate method for customizing the scope of variables:

protected IScope createLocalVarScope(IScope parentScope,
			LocalVariableScopeContext scopeContext) {

Take a look at the code of this method in XbaseScopeProvider, and you should get the general idea (the base implementation walks through the containment hierarchy of the model to build the scope for local variables); in particular, LocalVariableScopeContext contains the context for the scope. In our case, we need to provide a custom scope when the context is the greeting Model (in all other cases we rely on the base implementation):

package org.xtext.example.helloxvars.scoping;

import org.eclipse.emf.ecore.EObject;
import org.eclipse.xtext.scoping.IScope;
import org.eclipse.xtext.scoping.Scopes;
import org.eclipse.xtext.xbase.scoping.LocalVariableScopeContext;
import org.eclipse.xtext.xbase.scoping.XbaseScopeProvider;
import org.xtext.example.helloxvars.helloXvars.Model;

/**
 * This class contains custom scoping description.
 * 
 * see : http://www.eclipse.org/Xtext/documentation/latest/xtext.html#scoping on
 * how and when to use it
 * 
 */
public class HelloXvarsScopeProvider extends XbaseScopeProvider {

	@Override
	protected IScope createLocalVarScope(IScope parentScope,
			LocalVariableScopeContext scopeContext) {
		if (scopeContext != null && scopeContext.getContext() != null) {
			EObject context = scopeContext.getContext();
			if (context instanceof Model) {
				Model model = (Model) context;
				return Scopes.scopeFor(model.getVarDeclarations());
			}	
		}

		return super.createLocalVarScope(parentScope, scopeContext);
	}

}

So, when the context is the root of our model (Model) we return as the scope all the variables declared in our greeting model.

Now, restart your eclipse runtime instance and you’ll see that errors go away: the expressions in the greetings can access the variables declared above!

Note that the scoping does not concern only the expressions in the greetings, but also the initialization expressions of local variable declarations themselves:

val v1 = 'foo'
val v = new String(v1)
val v2 = v + v1

Now, if you do some experiments you’ll note that we provide scopes that contain too much! Try the following:

val v = new String(v2)
val v2 = v

This code should raise errors, since the initialization expression for v should not be able to access variables declared afterwards (e.g., v2); while, with our scope implementation, this code would be silently accepted.

So, we need to tweak our scope implementation; however, let’s stop following the workflow of trying a modification, restart the runtime eclipse instance, and see whether we made it… let’s be agile 🙂 and write some unit tests! Create an Xtend2 class in the tests plugin project org.xtext.example.helloxvars.tests and add in the MANIFEST org.eclipse.xtext.xtend2.lib as a dependency.

package org.xtext.example.helloxvars.tests

import com.google.inject.Inject
import org.eclipse.xtext.diagnostics.Diagnostic
import org.eclipse.xtext.junit4.InjectWith
import org.eclipse.xtext.junit4.XtextRunner
import org.eclipse.xtext.junit4.util.ParseHelper
import org.eclipse.xtext.junit4.validation.ValidationTestHelper
import org.eclipse.xtext.xbase.XbasePackage
import org.junit.Test
import org.junit.runner.RunWith
import org.xtext.example.helloxvars.HelloXvarsInjectorProvider
import org.xtext.example.helloxvars.helloXvars.Model

@InjectWith(typeof(HelloXvarsInjectorProvider))
@RunWith(typeof(XtextRunner))
class HelloXvarsParserTest {

	@Inject
	ParseHelper<Model> parser

	@Inject extension ValidationTestHelper

	@Test
	def void testParsingAndLinking() {
		parser.parse("Hello foo from new String()!").assertNoErrors
	}

	@Test
	def void testParsingAndLinkingWithVars() {
		parser.parse("
			val s1 = 'foo'
			val s2 = 'bar'
			val s3 = s1 + s2
			Hello foo from new String(s3)!
		").assertNoErrors
	}

	@Test
	def void testParsingAndLinkingWithClosures() {
		parser.parse("
			val s1 = 'foo'
			val s2 = 'bar'
			val s3 = s1 + [ s | s.toFirstLower + s1 ].apply(s2 + s1)
			Hello foo from new String(s3)!
		").assertNoErrors
	}

	@Test
	def void testParsingAndLinkingWithMissingVar() {
		parser.parse("
			Hello foo from new String(s)!
		").assertError(XbasePackage::eINSTANCE.XFeatureCall,
				Diagnostic::LINKING_DIAGNOSTIC,
				"Couldn't resolve reference to JvmIdentifiableElement 's'.")
	}

	@Test
	def void testParsingAndLinkingWithWrongVarOrder() {
		parser.parse("
			val s1 = s2
			val s2 = s1
			Hello foo from new String(s1)!
		").assertError(XbasePackage::eINSTANCE.XFeatureCall,
				Diagnostic::LINKING_DIAGNOSTIC,
				"Couldn't resolve reference to JvmIdentifiableElement 's2'.")
	}

}

You see, the first test checks a greeting program without variables, the second checks a valid program with variable access, the third one checks a valid program with variable access and a closure, the forth one checks that accessing a variable which is not declared generates an error, and the last one checks that variable initialization expressions cannot access variable declared afterwards. Of course, the last test fails! And we need to get the green bar back now 🙂

So now we modify the scoping:

package org.xtext.example.helloxvars.scoping;

import java.util.List;

import org.eclipse.emf.ecore.EObject;
import org.eclipse.xtext.EcoreUtil2;
import org.eclipse.xtext.scoping.IScope;
import org.eclipse.xtext.scoping.Scopes;
import org.eclipse.xtext.xbase.XExpression;
import org.eclipse.xtext.xbase.XVariableDeclaration;
import org.eclipse.xtext.xbase.scoping.LocalVariableScopeContext;
import org.eclipse.xtext.xbase.scoping.XbaseScopeProvider;
import org.xtext.example.helloxvars.helloXvars.Model;

/**
 * This class contains custom scoping description.
 * 
 * see : http://www.eclipse.org/Xtext/documentation/latest/xtext.html#scoping on
 * how and when to use it
 * 
 */
public class HelloXvarsScopeProvider extends XbaseScopeProvider {

	@Override
	protected IScope createLocalVarScope(IScope parentScope,
			LocalVariableScopeContext scopeContext) {
		if (scopeContext != null && scopeContext.getContext() != null) {
			EObject context = scopeContext.getContext();
			Model containingModel = EcoreUtil2.getContainerOfType(context,
					Model.class);
			XVariableDeclaration containingVarDecl = EcoreUtil2
					.getContainerOfType(context, XVariableDeclaration.class);
			List<XExpression> varDeclarations = containingModel.getVarDeclarations();
			int index = varDeclarations.size();
			if (containingVarDecl != null) {
				index = varDeclarations.indexOf(containingVarDecl);
			}
			return Scopes.scopeFor(varDeclarations.subList(0, index));
		}

		return super.createLocalVarScope(parentScope, scopeContext);
	}

}

This time we do something different: the scope will still be all the variables of the containing greeting Model; but if we’re inside a XVariableDeclaration, the scope will be the list of all the variable declarations up to the current variable, and we get the green line back… oh oh… something it’s still wrong: the previous failing test now succeeds, but the test for closure now fails! It says it cannot resolve the reference to ‘s’, the local variable of the closure itself:

	@Test
	def void testParsingAndLinkingWithClosures() {
		parser.parse("
			val s1 = 'foo'
			val s2 = 'bar'
			val s3 = s1 + [ s | s.toFirstLower + s1 ].apply(s2 + s1)
			Hello foo from new String(s3)!
		").assertNoErrors
	}

If you take a look at our implementation of scoping, you’ll soon realize that we customized it too much: we basically override completely the method of XbaseScopeProvider, which will then be unable to build the scope for local variables declared in the expressions, and closures declare local variable themselves!

Indeed, the right solution is something in between the two implementations of scoping we saw here:

1. we must check whether the container of the context is a greeting Model
2. if it is, then the scope is the list of variables up to the current one, if the context is a variable declaration (we can simply search for the context in the list of variable declarations, since List.indexOf accept any Object
3. otherwise, we delegate to XbaseScopeProvider

package org.xtext.example.helloxvars.scoping;

import java.util.List;

import org.eclipse.emf.ecore.EObject;
import org.eclipse.xtext.scoping.IScope;
import org.eclipse.xtext.scoping.Scopes;
import org.eclipse.xtext.xbase.XExpression;
import org.eclipse.xtext.xbase.scoping.LocalVariableScopeContext;
import org.eclipse.xtext.xbase.scoping.XbaseScopeProvider;
import org.xtext.example.helloxvars.helloXvars.Model;

/**
 * This class contains custom scoping description.
 * 
 * see : http://www.eclipse.org/Xtext/documentation/latest/xtext.html#scoping on
 * how and when to use it
 * 
 */
public class HelloXvarsScopeProvider extends XbaseScopeProvider {

	@Override
	protected IScope createLocalVarScope(IScope parentScope,
			LocalVariableScopeContext scopeContext) {
		if (scopeContext != null && scopeContext.getContext() != null) {
			EObject context = scopeContext.getContext();
			if (context.eContainer() instanceof Model) {
				Model model = (Model) context.eContainer();
				List<XExpression> varDeclarations = model.getVarDeclarations();
				int index = varDeclarations.indexOf(context);
				if (index < 0)
					index = varDeclarations.size();
				return Scopes.scopeFor(varDeclarations.subList(0, index));
			}
		}

		return super.createLocalVarScope(parentScope, scopeContext);
	}

}

We finally get the green line for our tests 🙂

Now let’s move on to generation! Similarly to the previous post about Xbase expressions, we will generate a main method for each Greeting, and we will generate the Java code for all the variable declarations and initializations by reusing the XbaseCompiler. Most of the code for the generation has already been explained in the previous post.

package org.xtext.example.helloxvars.generator

import org.eclipse.emf.ecore.resource.Resource
import org.eclipse.xtext.generator.IGenerator
import org.eclipse.xtext.generator.IFileSystemAccess
import com.google.inject.Inject
import org.eclipse.xtext.xbase.compiler.XbaseCompiler

import static extension org.eclipse.xtext.xbase.lib.IteratorExtensions.*
import org.xtext.example.helloxvars.helloXvars.Greeting
import org.eclipse.xtext.common.types.TypesFactory
import org.eclipse.xtext.xbase.compiler.ImportManager
import org.eclipse.xtext.xbase.XExpression
import org.eclipse.xtext.xbase.compiler.StringBuilderBasedAppendable
import org.xtext.example.helloxvars.helloXvars.Model
import org.eclipse.xtext.xbase.XFeatureCall

class HelloXvarsGenerator implements IGenerator {

	@Inject
	protected XbaseCompiler xbaseCompiler

	override void doGenerate(Resource resource, IFileSystemAccess fsa) {
		for(greeting: resource.allContents.toIterable.filter(typeof(Greeting))) {
			fsa.generateFile(
				greeting.packageName + "/" + // package
				greeting.className + ".java", // class name
				greeting.compile)
		}
	}

	def getJvmType(Greeting greeting) {
		val declaredType = TypesFactory::eINSTANCE.createJvmGenericType
		declaredType.setSimpleName(greeting.className)
		declaredType.setPackageName(greeting.packageName)
		declaredType
	}

	def compile(Greeting greeting) '''
	«val importManager = new ImportManager(true, getJvmType(greeting))»
	«val mainMethod = compile(greeting, importManager)»
	package «greeting.packageName»;
	«IF !importManager.imports.empty»

	«FOR i : importManager.imports»
		import «i»;
	«ENDFOR»
	«ENDIF»

	«mainMethod»
	'''

	def compile(Greeting greeting, ImportManager importManager) {
		var result = new StringBuilderBasedAppendable(importManager)
		for (varDecl : (greeting.eContainer as Model).varDeclarations) {
			result.append("\n// variable declaration")
			result = compile(varDecl, result)
		}
		result.append("\n// greeting expression")
		result = compile(greeting.expression, result)
		var expressionVar = result.getName(greeting.expression)
		if (greeting.expression instanceof XFeatureCall)
			expressionVar = result.getName((greeting.expression as XFeatureCall).feature)
		val compiled = '''
		public class «greeting.className» {
			public static void main(String args[]) {
				«result»

				Object expression = «expressionVar»;
				System.out.println("Hello «greeting.name» from " +
					expression.toString());
			}
		}
		'''
		compiled
	}

	def compile(XExpression xExpression, StringBuilderBasedAppendable result) {
		xbaseCompiler.toJavaStatement(xExpression, result, true)
		result
	}

	def className(Greeting greeting) {
		greeting.name.toFirstUpper
	}

	def packageName(Greeting greeting) {
		greeting.name.toLowerCase
	}
}

Let us concentrate on the interesting part

def compile(Greeting greeting, ImportManager importManager) {
	var result = new StringBuilderBasedAppendable(importManager)
	for (varDecl : (greeting.eContainer as Model).varDeclarations) {
		result.append("\n// variable declaration")
		result = compile(varDecl, result)
	}
	result.append("\n// greeting expression")
	result = compile(greeting.expression, result)
	var expressionVar = result.getName(greeting.expression)
	if (greeting.expression instanceof XFeatureCall)
		expressionVar = result.getName((greeting.expression as XFeatureCall).feature)
	val compiled = '''
	public class «greeting.className» {
		public static void main(String args[]) {
			«result»

			Object expression = «expressionVar»;
			System.out.println("Hello «greeting.name» from " +
				expression.toString());
		}
	}
	'''
	compiled
}

We basically use the XbaseCompiler for generating the Java code corresponding to variable declarations and initializations; then we need to access the generated Java variable name corresponding to the XExpression of the current Greeting element. We use the getName method, as explained in the previous post; however, this time, we need to distinguish the case when the XExpression is a XFeatureCall: this is the case when the XExpression is just the name of a declared variable, as in this code

var myList = 
	new LinkedList<Integer>(Collections::singleton(10))

Hello barList from myList !

In this case, we need to get the variable name which corresponds to the feature of the XFeatureCall, and not to the XFeatureCall itself (since no synthetic expression is generated for the XFeatureCall itself). Let’s try the generator

Remember, that we did not want to generate something meaningful: we wanted to experiment with code generation 🙂

You can find the sources for the project helloxvars at

https://github.com/LorenzoBettini/Xtext2-experiments

Hope you find this post useful, and stay tuned for new posts about Xtext 🙂