Code generation: Why so many steps?

Hi LaAck.
I had the same thought sometimes.

Noticing, that when you copy nodes selected in an editor, to clipboard, you get a textual representation – maybe it could be possible to [somehow] re-use the same code which handles the clipboard copy-to-text ?

Have you checked whether there is such code re-use in BaseLanguage (Java) generator?

I might face the same urge for re-use one day, but that would be for yaml-ish data schema language (editor and generator), in my case.


What project are you working on for your C# generation?

Are you also interested in generating multiple languages (a'la cross-platform) from the same concept instances? Currently I'm generating C++ and Java (classes, constructors, getters, setters, unmarshalling, etc.) from the same concepts of data schema language created with MPS...


So, what I also mean by all that is that we might have some overlap (e.g. generating non-Java in MPS; and a mindset of trying to avoid duplication), so let's keep in touch.


Wishing you success,
Karol Depka Pradzinski

I've also noticed your post http://forum.jetbrains.com/thread/Meta-Programming-System-993 about generating C# and XML from one model.

That seems quite similar to what I'm doing (Java and C++ files for every root node).
So let's keep in touch. Here is my LinkedIn: https://www.linkedin.com/in/karoldepka . I would be glad to know more about where you are from, what's your background, etc. :).

Best,
k

Hi Karol,

thank you for your reply and your suggestions.

Probably it would be possible to re-use the copy-paste implementation instead of writing the TextGen manually. Looking into the BaseLanguage implementation I noticed that they don't make use of the copy-paste handlers. They manually append the text snippets using a DSL (as always ;) )including some higher order functions.

The "project" I'm working on is kind of a dependency graph for parameters. Setting a parameter value effects re-calculation of depending parameters and so on. To build this graph I created a DSL - or, more precisely, I'm still working on it. ;) C# ist the target language for my code generation since the remaining parts of the system are specified in C#.

Yes, I'm interested in multiple-target generation. I need LaTeX, XML and C# generation on the long run. Maybe we have overlapping findings and we can share them. I'm currently working on different parts of the DSL. But the issue of code generation is still waiting for me. So, as you said, let's stay in touch.

Regards,
LaAck

Hi LaAck.
Thank you for your reply.
Please forgive my delay of replying. I was busy in other areas, but I will be intensifying my MPS work again soon.
I should have replied earlier anyway to keep the conversation going.
Now I'm up-to date with my email/forums again, so no such delay should occur.


I've also seen how the BaseLanguage generator is constructed under-the hood.
Actually the whole generator is, for me, the most troublesome part in practice. Although I like the paradigms that they are exploring there, in theory.

I also need (as an absolute requirement), multiple output files per root node: .cpp, .h, .java (which I have already), later .swift, maybe .yaml, maybe .html.
That's where the problems start with the built-in generator.
But this is not the only problem with the generator for me.

So, to circumvent the problems I've found, I've started developing an "alternative" generator, which supports multiple files output per root node. It also allows you to set file name to be arbitrary per-node. Whereas the built-in generator seems to force the file-name base part to be the same as the node name, which for me was a problem as well.

That "alternative generator" is using MPS build facets and generates to another directory, to avoid clashes with MPS build system.
Although the generator is more java-like and, you still have access to BaseLanguage&extensions and nice syntaxes for accessing MPS nodes.
You also have full control over formatting of the file text, file location.

If you are interested, I could share the code.
I could even imagine making it open source.
Although I'm not a big fan of "forks", but this generator fulfills use-cases that the original generator just apparently is not interested in (at least for now).
Plus, all the rest of MPS stays intact (editor, refactors, usage search, actions, etc.). Plus the built-in generator can co-exist with my generator, without any problem (although the output dir is different to avoid stepping on each other's toes).


If you will need LaTeX, C#, XML output, from single root node, then it is very likely that you will face the same problems as I faced (unless you find some other approach).

This is me - http://karoldepka.com/about/ , http://www.linkedin.com/in/karoldepka .

Let's stay in touch! I think MPS could use some more "community building" and awareness of "who is doing what".

Do you know more people who are using MPS or considering using it (apart from the MPS Forum users) ?
For example there is mbeddr, Markus Voelter, Fabien Campagne, .... We need more awareness of cases studies like those...

Best Wishes,
Karol Depka Pradzinski

Soon, I will also list all the problems that I see with the built-in generators (at least in my use-cases), and how I'm addressing them and why. With links to bugs that I've filed (and which ones were fixed or not fixed) and forum questions that I've posted and gotten replies to.
Best,
Karol

Hi Karol,

thank you again for your reply. "Business" is my daily situation, so I absolutely undestand the time-distribution issue, peering to my own delay of response. ;)

I absolutely agree with you regarding your opinion about the generator. The principles behind it are well-conceived and plausible. Nevertheless there should be a built-in mechanism for declaring source nodes to stay in the input AST. I'm really confused why this is not possible, yet, but in the case of the root node (root mapping rules) it IS possible using the property "keep input root". I suppose that it would be more desirable to be able to choose whether you want to use the reduction pattern or just a visitor pattern where the nodes are "visited" and USED for generation but they are not removed from the input AST.

As you supposed I encountered the same issue as you regarding the multiple-files-per-input-node topic. Currently I'm not able to generate both LaTeX and XML and that could be some kind of a show-stopper for MPS- :-/

So, yes, I'm really interested in your alternative generator implementation. I would be glad if you could share it. Nevertheless, though I'm sure your implementation does the job, I think it is essentially necessary to integrate the "forking" mechanism into the standard MPS generator soon (peering to the MPS developers ;) ). I think our use cases are only a very small subset of all possible use cases. So imho this is one of the most important missing features.

To stay in touch I've send you a connection invitation on LinkedIn. ;)

Yes, I know some more people who use MPS extensively but I'm not sure that they use the MPS forum. I also know Markus Voelter, Fabien Campagne (I've bought the two DSL books).

Thank you in advance and best wishes,
LaAck

Perhaps I am missing something. I understand your goal is to have an internal node of language A (i.e., a node of language A that is not a root node) generated to root nodes of language C,D and E.
Since you are able to specify that root nodes should not be removed from the input, couldn't you first generate the A node to one root node of some B language, then in language B define a generator that converts this node to a specific root node for C, D and E? (Then leave the generator of each C, D and E language generate to its output.)

I have not tried it, but we do generate multiple outputs for one root node in the NYoSh language generator. The only difference in our case is that the source node is a root node, so I assume if you first make a root node, the same mechanisms should work.  I would be interested to know the specifics if it does not work though.

Hi Fabien,

thank you very much for your reply. Actually we just want to transform our nodes of language A to nodes of e.g. languages B and C simultaneously. Let me give an example according to my use case:

Language A: DSL for creating parameter networks
Language B: DSL for creating LaTeX documents
Language C: DSL for creating XML documents

So when I use my language A to secify something like this...:

Parameter start = 1
Parameter end = 5
Parameter length = end - start

... I want to get one LaTeX document and one XML document from the same input specified using language A. As far as I know currently it is only possible to transform the input AST to nodes of ONE other language. Ok, there is one exception, i.e. the root node can be transformed multiple times if you enable the "keep input root" option. But is it possible to transform all internal nodes to nodes of two different languages simultaneously (Btw. when I say simultaneously I mean 'in one generation process'. It doesn't have to be done in parallel).

You said that you create multiple outputs for one root node on your NYoSh language generator. Do you only transform the root nodes or do you transform internal nodes as well? And if you do the latter HOW did you do it?

Thank you in advance!
LaAck

Hi LaAck :).
And Hi, Fabien :).

Just a quick reply, as I don't have much time this workweek...

@LaAck:
Thanks for adding me on LinkedIn.

Regarding "So, yes, I'm really interested in your alternative generator implementation. I would be glad if you could share it." - I'm eager to share it; first I would need to clean it up a bit to make it understandable and untangle it from stuff specific to my dayjob.
Are you sort-of blocked on that info now or is it ok if I send it on e.g. this Sunday?

I would put it on GitHub - is that okay?
That mechanism prototype I develop, currently works with a slightly older MPS version (3.0 or 3.1.0, not sure now) - is that ok for you?


(will reply more when I get more time).

Best,
Karol Depka Pradzinski

Btw, Fabien, are you on LinkedIn?
Best,
Karol

I think that a generator pre-processing script and the node.copy method can help. See this example: https://github.com/CampagneLaboratory/mps-multi-language-generation. After building solution_in_A you will need to look in the filesystem view to find the TextGen output.
It is not pretty, but it seems to work with existing mechanisms, suggesting that you could build a nicer abstraction on top of this technique.

Sorry it took me a while. We had a major deadline today. And no, I am not on linked in.

Hi all,

I tested the indicated repository and the Solution A does not rebuild. I got the messages below.
My MPS version is 3.1.4, I installed the TextOutput 1.2.1 .
Does anyone knows what I'm missing?

Thx,
Mar
consistency problem in dependencies map INPUT: [root] Root_D <no ref>[3552574974716278033] in Solution_in_A.model@0 current to original map: [root] Root <no ref>[9164034450995343220] in Solution_in_A.model@0 --> [root] Root <no ref>[9164034450995343220] in Solution_in_A.model consistency problem in dependencies map INPUT: [root] Root_E <no ref>[3552574974716278034] in Solution_in_A.model@0 current to original map: [root] Root <no ref>[9164034450995343220] in Solution_in_A.model@0 --> [root] Root <no ref>[9164034450995343220] in Solution_in_A.model

I tested with 3.1.5 from a fresh git clone and found I needed to remove the reference to language B, which I had used in earlier steps, but did not commit.

I have just pushed the change. This should fix the problem, if not let me know.

Hi Fabien,

in fact there were two errors one showed like red dashed lines into the editor and the other one (that I posted) showed at rebuild time. Probably the first was the one that was blocking generation.
Now generation is ok so when I changed the number of node into the Solution A I got a new version of rootA.e and rootA.d .

Now I'm fine thx, however FYI the warning red message (consistency problem in dependencies map) at rebuild time is still there.

Thx,
Mar

Hi Fabien,

I had some time to study your example however it's a bit confusing because there is a double configuration on how to generate the text output, or it seems to be double to a newbie like me.

For example it seems that one can configure the filename extension both in the TextGen (see Root_E_TextGen) and into the generator (see map_Root_E), in fact the ladder seems to don't have effect and the output file is influenced only by the TextGen aspect.

However this post requires to generate multiple output from one concept (root A) and in this sense your example it's perfect. But I asked to myself why TextOutput is there, if it has a (obscure to me) reason or if you just forgot to clean up to code.

In any case I found your example very usefull.
Thx,
Mar

Hi,

The above mentioned example https://github.com/CampagneLaboratory/mps-multi-language-generation does not open in the current version of MPS. Why wasn't it migrated automatically? The support for backward compatibility(or lack of it rather) looks scarry :-/

If someone is still interested in the solution of the original question, see this:

https://mps-support.jetbrains.com/hc/en-us/community/posts/360004619119-Copy-Action-in-MPS-Ctrl-C-