The JvmModelInferrer, the debugger and Xbase in Xtext 2.3

In previous posts, I had blogged about some experiments done with Xtext 2 wonderful mechanisms to access Java types and to reuse Xbase expressions; however, when generating code, I was using the “manual way”. That is to say, I was not using the JvmModelInferrer mechanism; the mechanism of JvmModelInferrer allows to define a mapping from your DSL elements to Java elements. You’ll then have for free a complete integration of Java into your DSL, and viceversa (yes, from Java you can jump directly to your DSL elements starting from generated classes, methods, etc.). And by the way, you’ll also have for free Java code generation (corresponding to the mapping you defined in your implementation of JvmModelInferrer).

When I wrote those posts, I had done some experiments to implement my DSL Xsemantics, http://xsemantics.sourceforge.net/about/ ; I was reluctant to use the JvmModelInferrer because the elements of Xsemantics could not be mapped directly into Java concepts (actually into element of the Jvm model of Xtext), and all the examples I had seen around using the JvmModelInferrer (e.g., the Domainmodel example shipped with Xtext itself) had a very direct mapping to Java concepts. Thus I preferred to learn how to manually generate Java code using Xtend and ImportManager functionalities.

Now, Xtext 2.3 is about to be released, and I started to try the latest milestones; the JvmModelInferrer mechanisms have been improved and one of the coolest new features of Xtext 2.3 is that by using JvmModelInferrer you also get for free debugging functionalities for your DSL (yes, you read correctly, you can debug your DSL programs source code while debugging the Java code!!!). So I decided to try to learn how to use JvmModelInferrer; I’ve just started to port Xsemantics to use JvmModelInferrer, but in the meantime I’ve done some experiments with a simple hello DSL, and I’d like to share my experiences here 🙂

This hello DSL we use here has some features that require some customizations in order to coexist with Xbase; in particular, when using the JvmModelInferrer to define the mapping most of scoping and validation of Xbase will work out of the box; however, we will have to customize something for our DSL. In particular we will have to

1. tweak the XbaseValidator to deal with some situations which are
   1. correct in our language but not in Xbase
   2. correct in Xbase but not in our language
2. tweak the XbaseScopeProvider to make something visible in our code
3. implement a non direct mapping to the Jvm model in the JvmModelInferrer

Remember that you need the milestone update site for Xtext 2.3, until it is released on June:

http://download.eclipse.org/modeling/tmf/xtext/updates/composite/milestones/

The HelloInferrer language

The idea of this DSL (called HelloInferrer) is that you can define Hello elements (which will correspond to Java classes) with some operations which are NOT directly mapped to Java methods: in particular, each operation consists of

1. a name
2. a (possibly empty) list of (input) parameters (which corresponds to Java parameters)
3. an output parameter
4. a body

In the body

1. the output parameter corresponds to a local variable that can be assigned;
2. no return statement is allowed inside the body;
3. the return value in the corresponding Java method will be a wrapper around the output parameter;
4. the wrapper is an instance of a library class HelloResult, with a type parameter which corresponds to the Java type of the output parameter.

It’s probably easier to show an example of an input program…

Hello my.hello.MyHello {
	op myOp(int i, boolean b) output String s {
		if (b)
			s = "b is true"
	}
}

… and the corresponding expected generated Java code

import org.xtext.example.helloinferrer.runtime.HelloResult;

public class MyHello {
  public HelloResult<String> myOp(final int i, final boolean b) {
    String s = null; // output parameter
    if (b) {
      s = "b is true";
    }
    return new HelloResult<String>(s);
  }
}

What’s this DSL for? Nothing! It’s just to experiment with JvmModelInferrer (and the rules you define in Xsemantics have similar functionalities).

I will try to detail the steps I followed to get to the result, of course, in a test driven fashion 🙂 In particular, we will write a class for tests related to validations and another one for tests related to code generation.

First, this is the Xtext grammar for our language

grammar org.xtext.example.helloinferrer.HelloInferrer with 
	org.eclipse.xtext.xbase.Xbase

generate helloInferrer "http://www.xtext.org/example/helloinferrer/HelloInferrer"

Model:
	imports += Import*
	greetings+=Greeting*;

Import:
  'import' importedNamespace = QualifiedNameWithWildcard
;

QualifiedNameWithWildcard:
  QualifiedName '.*'?
; 

Greeting:
    'Hello' name=QualifiedName '{'
    	operations += Operation*
    '}'
;

Operation:
	'op' name=ValidID '(' 
		(params+=FullJvmFormalParameter (',' params+=FullJvmFormalParameter)*)? 
	')' 'output' output=FullJvmFormalParameter
	body=XBlockExpression
;

First, let’s start to implement the inferrer at least for the class corresponding to an Hello element. We expect the following Java code for the given Hello definition (we wrote the test in Xtend2, using Xtext unit test functionalities, in particular the new CompilationTestHelper, which provides useful features):

class HelloInferrerCompilerTest {

	@Inject extension CompilationTestHelper

	@Test
	def void testGeneratedJavaClass() {
		'''
		Hello my.test.MyHello {

		}
		'''.assertCompilesTo(
		'''
		package my.test;

		public class MyHello {
		}
		'''
		)
	}

}

The first version of the inferrer to make this test pass is the following:

class HelloInferrerJvmModelInferrer extends AbstractModelInferrer {

    /**
     * convenience API to build and initialize JVM types and their members.
     */
	@Inject extension JvmTypesBuilder

	@Inject extension IQualifiedNameProvider

   	def dispatch void infer(Greeting element, IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
   		acceptor.accept(
			element.toClass( element.fullyQualifiedName )
		).initializeLater [
			documentation = element.documentation
		]
   	}
}

Now let’s write a test for the parsing of a HelloInferrer program which also defines an operation with a body accessing the input parameters:

@RunWith(typeof(XtextRunner))
@InjectWith(typeof(HelloInferrerInjectorProvider))
class HelloInferrerParserTest {

	@Inject extension ParseHelper<Model>

	@Inject extension ValidationTestHelper

	@Test
	def void testOperation() {
		'''
		Hello my.test.hello {
			op myOp(String s, int i) output Boolean b {
				val foo = s + i
			}
		}
		'''.checkModel
	}

	def checkModel(CharSequence prog) {
		val model = prog.parse
		Assert::assertNotNull(model)
		model.assertNoErrors
		model
	}

}

This test will fail since in the body of the operation the parameters cannot be accessed. Instead of defining a custom scoping we use the JvmModelInferrer mechanism to map the operation to a Java method (we map each Operation’s parameter to a parameter in this Java method, and the Operation’s body to the body of the Java method). This will be enough to make the XbaseScopeProvider happy (for the moment) since it will be able to implement scoping correctly!

For the moment we map an Operation to a void method; then later we will deal with the result of the operation.

   	def dispatch void infer(Greeting element, IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
   		acceptor.accept(
			element.toClass( element.fullyQualifiedName )
		).initializeLater [
			documentation = element.documentation
			for (o : element.operations) {
				members += o.toMethod(o.name, null) [
					documentation = o.documentation
					for (p : o.params) {
						parameters += p.toParameter(p.name, p.parameterType)
					}
					body = o.body
				]
			}
		]
   	}

The parser test now succeeds, since the parameters can be found.

Now let’s see what we get during generation:

	@Test
	def void testGeneratedJavaMethod() {
		'''
		Hello my.test.MyHello {
			op myOp(String s, int i) output Boolean b {
				val foo = s + i
			}
		}
		'''.assertCompilesTo(
'''
package my.test;

public class MyHello {
  public void myOp(final String s, final int i) {
    final String foo = (s + Integer.valueOf(i));
    return null;
  }
}
'''
		)
	}

Now the generated Java code looks suspicious: we have a void method which returns something… this should not compile.

The new framework for testing code generation in Xtext 2.3 provides easy means to test whether the generated Java code actually compiles in Java; look at the CompilationTestHelper API and you’ll see. We use these API as follows:

	@Test
	def void testGeneratedJavaMethod() {
		'''
		Hello my.test.MyHello {
			op myOp(String s, int i) output Boolean b {
				val foo = s + i
			}
		}
		'''.compile[assertEquals(
'''
package my.test;

public class MyHello {
  public void myOp(final String s, final int i) {
    final String foo = (s + Integer.valueOf(i));
    return null;
  }
}
'''.toString, generatedCode
		)
		compiledClass
		]
	}

The compile method will try to parse and validate the given source and then will invoke the generator on it; the passed closure can then access the result; in particular compiledClass will try to compile (in Java) the corresponding generated code and if it succeeds it will return the Java Class. If you now run this test it will fail since the generated code is not correct Java code. This is due to the fact that in the inferrer we specified null as the return type of the method, while we should specify a type representing void, here’s the correct inferrer:

...
	@Inject extension TypeReferences
...

   	def dispatch void infer(Greeting element, IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
   		acceptor.accept(
			element.toClass( element.fullyQualifiedName )
		).initializeLater [
			documentation = element.documentation
			for (o : element.operations) {
				members += o.toMethod(o.name, "void".getTypeForName(o)) [
					documentation = o.documentation
					for (p : o.params) {
						parameters += p.toParameter(p.name, p.parameterType)
					}
					body = o.body
				]
			}
		]
   	}

Now, we fix the test with the corrected expected code; and since we will use this testing strategy for the generated code in other tests, we create a reusable method in the test class

	@Test
	def void testGeneratedJavaMethod() {
'''
Hello my.test.MyHello {
	op myOp(String s, int i) output Boolean b {
		val foo = s + i
	}
}
'''.assertCorrectJavaCodeGeneration(
'''
package my.test;

public class MyHello {
  public void myOp(final String s, final int i) {
    final String foo = (s + Integer.valueOf(i));
  }
}
''')
	}

	def private assertCorrectJavaCodeGeneration(CharSequence input, CharSequence expected) {
		input.compile [
			// check the expected Java code
			assertEquals(expected.toString, generatedCode)
			// this will issue Java generation
			compiledClass
		]
	}

Dealing with output parameter

We would like to assign values to the output parameter; however, since it is not put in the parameters of the inferred method for the operation, this test will fail with a link error, since the output parameter cannot be referred in the body of the operation

	@Test
	def void testAccessToOutput() {
		'''
		Hello my.test.hello {
			op myOp(String s, int i) output Boolean b {
				b != null
			}
		}
		'''.checkModel
	}

We thus need to create a custom scope provider (extending XbaseScopeProvider)

public class HelloInferrerScopeProvider extends XbaseScopeProvider {

	@Inject
	IJvmModelAssociations associations;

	@Override
	protected IScope createLocalVarScopeForJvmOperation(JvmOperation context,
			IScope parentScope) {
		parentScope = super.createLocalVarScopeForJvmOperation(context,
				parentScope);

		// retrieve the AST element associated to the method
		// created by our model inferrer
		EObject sourceElement = associations.getPrimarySourceElement(context);
		if (sourceElement instanceof Operation) {
			Operation operation = (Operation) sourceElement;
			return createLocalScopeForParameter(operation.getOutput(),
					parentScope);
		}

		return parentScope;
	}
}

What is going on here? Well, since in our model inferrer we created a method, through the JvmTypesBuilder.toMethod (actually, a JvmOperation), for an Operation, the JvmTypesBuilder associated the Operation in the AST model to the created JvmOperation. The scope provider will not compute the scope for the original Operation: it will compute it for the associated JvmOperation, that is why we redefine the method createLocalVarScopeForJvmOperation. However, in our custom implementation we need the original Operation… we can retrieve it using an injected IJvmModelAssociations!

The previous test now passes.

Now we would like to be able to assign to the output parameter, but this test will fail

	@Test
	def void testAssignToOutput() {
		'''
		Hello my.test.hello {
			op myOp(String s, int i) output Boolean b {
				b = (s != null)
			}
		}
		'''.checkModel
	}

Since the Xbase validator considers by default all parameters (JvmFormalParameter) as final. Thus, we must intercept this checking in the validator, and do not generate errors if we are trying to assign to an Operation’s output parameter:

@SuppressWarnings("restriction")
public class HelloInferrerJavaValidator extends
		AbstractHelloInferrerJavaValidator {

	@Override
	public void checkAssignment(XAssignment assignment) {
		JvmIdentifiableElement assignmentFeature = assignment.getFeature();
		if (assignmentFeature instanceof JvmFormalParameter) {
			EObject container = assignmentFeature.eContainer();
			// it is OK to assign to an Operation's output parameter
			if (container instanceof Operation
					&& ((Operation) container).getOutput() == assignmentFeature) {
				return;
			}
		}
		super.checkAssignment(assignment);
	}
}

Note that the method we redefine is in XbaseJavaValidator, and our (generated) AbstractHelloInferrerJavaValidator extends XbaseJavaValidator, since our grammar reuses Xbase grammar. Now the above test succeeds.

However, let’s also make sure that we did not break anything, and that standard input parameters cannot be assigned, and that when we assign to an output parameter the type checking still works:

	@Test
	def void testWrongAssignToInputParam() {
		'''
		Hello my.test.hello {
			op myOp(String s, int i) output Boolean b {
				s = null
			}
		}
		'''.parse.assertError(
			XbasePackage::eINSTANCE.XAssignment,
			IssueCodes::ASSIGNMENT_TO_FINAL
		)
	}

	@Test
	def void testWrongAssignToOutputParam() {
		'''
		Hello my.test.hello {
			op myOp(String s, int i) output Boolean b {
				b = s
			}
		}
		'''.parse.assertError(
			XbasePackage::eINSTANCE.XFeatureCall,
			IssueCodes::INCOMPATIBLE_TYPES
		)
	}

Generation for Output Parameter

Now we want to deal with output parameter in the generation. If we try to generate code from the following source

Hello my.test.MyHello {
	op myOp(String s, int i) output Boolean b {
		b = true
	}
}

We’ll get an exception during generation, since the variable for the output parameter cannot be found.

From now on, things start to get more interesting in the JvmModelInferrer 🙂 Here’s how we modify it

class HelloInferrerJvmModelInferrer extends AbstractModelInferrer {
...	
	@Inject extension TypeReferenceSerializer

	@Inject XbaseCompiler xbaseCompiler

   	def dispatch void infer(Greeting element, IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
   		acceptor.accept(
			element.toClass( element.fullyQualifiedName )
		).initializeLater [
			documentation = element.documentation
			for (o : element.operations) {
				members += o.toMethod(o.name, "void".getTypeForName(o)) [
					documentation = o.documentation
					for (p : o.params) {
						parameters += p.toParameter(p.name, p.parameterType)
					}
					body = [
						it.declareVariableForOutputParameter(o.output)

						xbaseCompiler.compile(o.body, it, "void".getTypeForName(o), null)
					]
				]
			}
		]
   	}

   	def declareVariableForOutputParameter(ITreeAppendable it, JvmFormalParameter o) {
   		val outputVarName = it.declareVariable(o, o.simpleName)
		o.parameterType.serialize(o, it)
		it.append(" " + outputVarName + " = null; // output parameter")
   	}
}

Let’s see what’s going on here:

Instead of assigning to the body of the mapped method the body of the operation, we assign a closure; this closure is automatically passed a ITreeAppendable (a new appendable introduced in Xtext 2.3 which is used also for the debugging functionalities). The nice thing is that when this closure will be executed the passed appendable will already be configured with the binding for this (and if our language had superclasses also super would be bound) which will then work out of the box in your Operation’s method and in the generated Java code) – you might want to take a look at JvmModelGenerator.xtend::createAppendable, which is part of xbase plugin. Furthermore, since this is the body of a mapped method, the appendable which is passed to our closure will also have the bindings for the Java method’s parameters (which were associated to our Operation’s input parameters).

Then, we need to use this appendable to generate the Java declaration of the local variable which corresponds to our Operation’s output parameter:

1. we thus declare a variable in the appendable; this variable is associated (in the appendable) to our output parameter (this will only map our output parameter to a name in the appendable, which will be used internally when the xbase compiler needs to refer to our output parameter: it will not generate anything in the output buffer);
2. we use the TypeReferenceSerializer to generate in the Java code for the type of the local variable for our output parameter (this will also deal with possible imports in the generated Java code, since it relies on the ImportManager);
3. we then generate the name of the local variable (using the one declared above into the appendable) and its initialization to null.

Then, we rely on an injected XbaseCompiler to generate the Java code corresponding to the Operation’s body. Since we declared a variable for the output parameter in the appendable, if the original body referred to the output parameter, the xbase compiler will be able to generate code for the use of output parameter!

Note that we initialize the generated variable for the output parameter to null; this works only if the output parameter’s type is NOT a basic (or primitive) type, e.g. int or boolean. To keep things simple we are not dealing with primitive types for output parameters (this will simplify also the generated return type). Thus, we also must make sure to rule out such use, with a validator’s check method:

import org.eclipse.xtext.common.types.util.Primitives;

...
	@Inject
	private Primitives primitives;

	@Check
	public void checkOperation(Operation operation) {
		if (operation.getOutput() != null
			&& operation.getOutput().getParameterType() != null) {
			if (primitives
					.isPrimitive(operation.getOutput().getParameterType())) {
				error("Primitive types cannot be used as output parameters.",
						operation.getOutput().getParameterType(), null,
						IssueCodes.INVALID_USE_OF_TYPE);
			}
		}
	}

Note that we rule out primitive types ONLY for output parameters. Of course, we test this as well

	@Test
	def void testInvalidPrimitiveType() {
		'''
		Hello my.test.hello {
			op myOp(String s, int i) output boolean b {
				return true
			}
		}
		'''.parse.assertError(
			TypesPackage::eINSTANCE.jvmTypeReference,
			IssueCodes::INVALID_USE_OF_TYPE,
			"Primitive types cannot be used as output parameters."
		)
	}

Dealing with generated method return type

Let’s add a class that will be used for returning the result of generated Java methods

package org.xtext.example.helloinferrer.runtime;

public class HelloResult<T> {
	T value;

	public HelloResult(T value) {
		super();
		this.value = value;
	}

	public T getValue() {
		return value;
	}

	public void setValue(T value) {
		this.value = value;
	}

}

We now change the return type of the inferred method as follows

members += o.toMethod(o.name, 
	typeof(HelloResult).
		getTypeForName(o, o.output.parameterType))

Most tests will fail, due to Xbase validation issues; for instance, for this test program

Hello my.test.hello {
	op myOp(String s, int i) output Boolean b {
		val foo = s + i
	}
}

You’ll get this validation error

java.lang.AssertionError: Expected no errors, but got :[ERROR:Incompatible implicit return type. Expected org.xtext.example.helloinferrer.runtime.HelloResult<java.lang.Boolean> but was void (__synthetic0.helloinferrer line : 2)]

Since the block is expected to return a value of type HelloResult<Boolean> (because it is contained in an inferred Java method which declares to return such type) while it does not return anything. We will need to generate the actual return statement in the Java code, but first we must also avoid that Xbase validator complains about that, thus in our validator we redefine the appropriate method:

	@Override
	public void checkImplicitReturn(XExpression expr) {
		// we will deal with this during generation
		return;
	}

	@Override
	public void checkReturn(XReturnExpression expr) {
		error("Explicit return not available in this language.", expr, null,
				ValidationMessageAcceptor.INSIGNIFICANT_INDEX, INVALID_RETURN);
	}

Note that we also make sure that in our programs explicit return expressions are not allowed (since in our DSL return statements do not make sense), and of course we test it

	@Test
	def void testInvalidReturn() {
		'''
		Hello my.test.hello {
			op myOp(String s, int i) output Boolean b {
				return true
			}
		}
		'''.parse.assertError(
			XbasePackage::eINSTANCE.XReturnExpression,
			IssueCodes::INVALID_RETURN,
			"Explicit return not available in this language."
		)
	}

We now modify the inferrer so that it also generates the final Java return statement (according to the generated method’s return type) creating a new HelloResult (with the type of the Operation’s output parameter as the type argument) and passing as the argument the output parameter:

   	def dispatch void infer(Greeting element, IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
   		acceptor.accept(
			element.toClass( element.fullyQualifiedName )
		).initializeLater [
			documentation = element.documentation
			for (o : element.operations) {
				members += o.toMethod(o.name, 
					o.output.returnType
				) [
					documentation = o.documentation
					for (p : o.params) {
						parameters += p.toParameter(p.name, p.parameterType)
					}
					body = [
						it.declareVariableForOutputParameter(o.output)

						xbaseCompiler.compile(o.body, it, "void".getTypeForName(o), null)

						it.generateFinalReturnStatement(o.output)
					]
				]
			}
		]
   	}

   	def declareVariableForOutputParameter(ITreeAppendable it, JvmFormalParameter o) {
   		val outputVarName = it.declareVariable(o, o.simpleName)
		o.parameterType.serialize(o, it)
		it.append(" " + outputVarName + " = null; // output parameter")
   	}

   	def returnType(JvmFormalParameter o) {
   		if (o != null && o.parameterType != null)
   			typeof(HelloResult).getTypeForName(o, o.parameterType)
   	}

   	def generateFinalReturnStatement(ITreeAppendable it, JvmFormalParameter o) {
   		it.newLine.append("return new ")
   		o.returnType.serialize(o, it)
   		it.append('''(«it.getName(o)»);''')
   	}

Note that we check, when generating the return type, that the JvmFormalParameter is not null and also that the parameterType feature is not null; we need to check this because when writing a program in our DSL editor the inferrer is used also with a possibly not finished program and thus the JvmFormalParameter or its features can be null, for instance:

op foo(String s) // the JvmFormalParameter is null
op foo(String s) output // its parameterType feature is null

We can now test that Java code is correctly generated

'''
Hello my.test.MyHello {
	op myOp(String s, int i) output Boolean b {
		myOp2(true, 'foo')
		this.myOp2(true, 'foo')
	}

	op myOp2(boolean b, String s) output Integer i {

	}
}
'''.assertCorrectJavaCodeGeneration(
'''
package my.test;

import org.xtext.example.helloinferrer.runtime.HelloResult;

public class MyHello {
  public HelloResult<Boolean> myOp(final String s, final int i) {
    Boolean b = null; // output parameter
    this.myOp2(true, "foo");
    this.myOp2(true, "foo");
    return new HelloResult<Boolean>(b);
  }

  public HelloResult<Integer> myOp2(final boolean b, final String s) {
    Integer i = null; // output parameter
    return new HelloResult<Integer>(i);
  }
}
''')

We also do another test:

	@Test
	def void testAutomaticBoxing() {
'''
Hello my.test.MyHello {
	op myOp(String s, int j) output Integer i {
		i = j + 1
		if (i < 0)
			i = 1
	}
}
'''.assertCorrectJavaCodeGeneration(
'''
package my.test;

import org.xtext.example.helloinferrer.runtime.HelloResult;

public class MyHello {
  public HelloResult<Integer> myOp(final String s, final int j) {
    Integer i = null; // output parameter
    int _plus = (j + 1);
    i = Integer.valueOf(_plus);
    boolean _lessThan = ((i).intValue() < 0);
    if (_lessThan) {
      i = Integer.valueOf(1);
    }
    return new HelloResult<Integer>(i);
  }
}
''')
	}

The result of this test shows an interesting thing: the type system of Xbase (and its generator) automatically performs boxing and unboxing for primitive types!

Another interesting test is the following:

	@Test
	def void testUseOfResult() {
'''
Hello MyHello {

	op foo(int j) output Integer i {
		i = new MyHello().bar(this).value.bar2().value
	}

	op bar(MyHello myHello) output MyHello result {
		result = myHello
	}

	op bar2() output Integer res {
		res = 10
	}
}
'''.assertCorrectJavaCodeGeneration(
'''
import org.xtext.example.helloinferrer.runtime.HelloResult;

public class MyHello {
  public HelloResult<Integer> foo(final int j) {
    Integer i = null; // output parameter
    MyHello _myHello = new MyHello();
    HelloResult<MyHello> _bar = _myHello.bar(this);
    MyHello _value = _bar.getValue();
    HelloResult<Integer> _bar2 = _value.bar2();
    Integer _value_1 = _bar2.getValue();
    i = _value_1;
    return new HelloResult<Integer>(i);
  }

  public HelloResult<MyHello> bar(final MyHello myHello) {
    MyHello result = null; // output parameter
    result = myHello;
    return new HelloResult<MyHello>(result);
  }

  public HelloResult<Integer> bar2() {
    Integer res = null; // output parameter
    res = Integer.valueOf(10);
    return new HelloResult<Integer>(res);
  }
}
''')
	}

This shows that, since the inferrer provides a “Java view” of your Greeting element and of its operations, you can already use the inferred Java elements in the Operations’ bodies: since the Operation bar’s output parameter is a MyHello, in Java this corresponds to a method returning a HelloResult<MyHello>, thus in

new MyHello().bar(this).value

“value” corresponds to invoking “getValue” on the object (of type HelloResult<MyHello>) returned by the Java method inferred for bar.

Running the editor

In the implementation of the DSL there’s also a project wizard (Helloinferrer project) which creates a plugin project with an example MyHello.helloinferrer, and a Java file, MyHelloMain.java that uses the Java code generated from MyHello.helloinferrer. You can see, thanks to the powerful Xbase mechanisms (tightly integrated with Java), that you can use involved Java types, and all Xbase expression syntax, like closures. In the screenshot you can see the example program (on the left), the generated Java code (on the right) and an example of Java main program using it (bottom).

Dealing with Debugging features

One of the coolest new features in Xtext 2.3 is the debugging functionalities which you get when using Xbase expressions: if you debug an application which uses the Java code generated from your DSL programs, then you can debug the original sources instead of the generated Java code! You can try that by setting breakpoints in your DSL program sources (the context menu breakpoint items are still missing, but you can double click the breakpoint ruler):

DISCLAIMER: what follows is just my interpretation of the debugging mechanisms of the TreeAppendable (I found no explicit documentation); debugging works with the following modified code, but I might have gotten the semantics of the trace API wrong 🙂

For the Java parts that are generated with the XbaseCompiler in our JvmModelInferrer, the debugging features already work! However, if you try to enter into one of the Operations (with Step Into) you will see that in the stack trace you get some “unknown locations” error, and the debugging gets back to work if you further press Step Into (getting to the first line of the Operation). This is due to the generated variable declaration for the output parameter.

We then must modify the inferrer so that it uses the passed appendable to trace the generation of the declaration of the Java local variable corresponding to the output parameter (and also the final generated Java return statement).

   	def declareVariableForOutputParameter(ITreeAppendable it, JvmFormalParameter o) {
   		val outputVarName = it.declareVariable(o, o.simpleName)
   		val childAppendable = it.trace(o, true)
		o.parameterType.serialize(o, childAppendable)
		childAppendable.append(" " + outputVarName + " = null; // output parameter")
   	}

   	def generateFinalReturnStatement(ITreeAppendable it, JvmFormalParameter o) {
   		val childAppendable = it.trace(o, false)
   		childAppendable.newLine.append("return new ")
   		o.returnType.serialize(o, childAppendable)
   		childAppendable.append('''(«childAppendable.getName(o)»);''')
   	}

Note that when declaring the local variable we specify true as the second trace argument since we want that to be used during debugging. We don’t need that when generating the final return statement (and we pass false).

Now, create a Helloinferrer project with the wizard, and start debugging the MyHelloMain.java as a Java application:

When you get to the line invoking printMappedList (which corresponds to an Operation in MyHello.helloinferrer), Step Into and see the magic:

You see! You’re debugging the original source!

You can try to Step Over to pass to the next line

or try Resume so that the debugger stops at the breakpoint which is inside the closure (and note the value of the variable it in the Variables view)

Try Resume again, and the debugger will hit the breakpoint again (and see the changed value for it in the Variables view):

Getting the sources

The sources can be retrieved by this git repository:

https://github.com/LorenzoBettini/Xtext2-experiments

and the project is helloinferrer.