DSL Testing


currently I'm working on test automation for DSLs. Referring to MPS there are some aspects for which some kind of unit tests could be useful. Just to give some examples:

  • I specified a constraint which should make sure that only children of a particular concept can be inserted. I could test this by creating a model and instantiating the respective parent node and then use the auto-completion menu. But if I have to make sure that everything has been tested, it would be more useful to go through a collection of programmatic aspect tests.
  • How can I test programmatically if my TextGen works as believed? (This question comes from: Text generator unit tests)
  • One could also write migration scripts. How can I make sure programmatically that the transformation from an old AST to the new one works as desired?

There are more examples of leverage points of testing. I know that there is a test aspect. But it seemingly does not cover all aspects. Is there any best practice or tutorial on how to write such tests?

Thank you in advance,
The testing depends on what you test.
  • constraints: while it's not possible to check completion menu, still if you want to test your constraining code (not how MPS completion menu works), you can make a test with a "bad" model and check whether MPS reports some broken constraints (like the one on the screenshot, screen is from 3.2 EAP)
  • textgen: I think the most reasonable way to check textgen is by generating models for which we already know correct generation result, like said in the topic you've referenced.
  • migration: you can do it just by extracting a method that migrates one node in the migration and using it in a "node test case", for example.

In common, you are right that now there's a problem with the whole system: there's no "universal" way to test any aspect and no tutorial.  We'll think about what we could do here. If you have any ideas on that, don't hesitate to propose any.
You may have found http://confluence.jetbrains.com/display/MPSD31/Testing+languages already. This is all there is at the moment to testing in MPS. We will try to add more details on testing.
All tests are performed within your browser over HTTP (TCP) and require no further software other than a newer browser with activated Flash.
Thank you for your reply and sorry for my delayed reply!

I absolutely agree with you referring to your statement about the dependency between "what to test" and "how to test". Referring to your examples, I suggest the following:
  • constraints: I do not want to test whether the MPS code completion menu works as desired. Instead I've been thinking of some kind of test language to check if my constraints work. By just creating a "bad" model one would encounter the issue that he or she has to remember the constraints in the model. Using a dedicated test language it would be possible to explicitly write down the expectations the language author has. Additionally the author could run all test cases for the language and it would be possible to check language correctness at a glance. Of course the latter depends on the completeness of your test cases. Stepping over the different statements of a constraint while executing the test would also be very helpful.
  • textgen: It is almost the same argumentation as for the constraints aspect. By providing a test case that checks whether the textgen transforms node<PublicVisibility> to "public" one could again provide expectations. The notation could be very easy, e.g.
node<PublicVisibility> yields "public"
condition: <<always>>

Maybe there is a more expressive or useful notation.
  • migration: Ok, this would be possible, I think.

If I can help with some requirements for testing, don't hestitate to ask me. I'm only one MPS user but my requirements may be a useful starting point.

Thank you again and kind regards,
Yes, thank you. I already found this documentation. It is helpful but not to the extend, I currently need.
Could you provide more details to explain what you mean, please? I don't get it, yet.

Please sign in to leave a comment.