Referencing to a class generated in another language/generator

Anyone? This is still unresolved.

How do I reference from one generator to classes generated in another generator?

Hi Dmitry,

i think http://code.google.com/p/mps-lwc11/downloads/detail?name=LWC11-MPS-Docs.pdf&can=2&q=

Page 36ff will help you ...

Best,
Dan

Dan, thanks a lot, very interesting reading. Will what they're suggesting it tomorrow.

Funny, I actually came here to ask this exact question to see the answer at the top of the list. I actually remember reading that passage in that tutorial when I was getting going w/ all this and not understanding what it was getting at, but now I do.

Wowzers on the solution though :p

On my Mac using 2.0.2 I'm modifying /Applications/MPS 2.0.app/bin/mps.vmoptions with "-Dmps.internal=true" and I'm still not seeing the "Stubs for Generated Code" checkbox as described in the tutorial. I've modified all of my references to link by name, and things continue to work if a module is self-contained, but if I start referencing things in other modules the references aren't getting hooked up. Is that checkbox still necessary? If so am I doing something wrong to get it to show up?

Did you have any luck? I'm trying a similar thing.

I wasn't lucky too. Solved my problem by redesigning generated classes to be independent on variable names.

No, it didn't work for me. I decided to dump all my concepts in one single module so far. But that's just temporary solution.

Hrm, I don't think that's possible for my language. Did you even get the checkbox to show up?

I put together a simple project demonstrating the problem. You can build the "sandbox" module just fine, but if you try to build "sandbox2" which references things in "sandbox" you get errors. I have '-Dmps.internal=true' in my vmoptions as specified above and do not see the 'Stubs for Generated Code' checkbox in the advanced tab of my solution properties. Can you advise us how we could make the "sandbox2" module build?
CrossModuleReferenceTest.zip (74KB)

Btw, interestingly enough I can use "Preview Generated Text" and it produces the right stuff, but when I make it I get errors about not being able to hook up references

Ok,

i m using the 1.5 version of mps. There i can set the mps.internal=true and i can select the generate stup for solution checkbox. Like described in the lwc11 guide, i can reference generated classes from other models then ...

Dan

No, I don't see the checkbox. I do see new checkbox "Load classes with app classloader", so I know the option is in effect.

Huh, it sounds like "support" for this technique may have been broken in recent versions, I posted a reply in http://forum.jetbrains.com/thread/Meta-Programming-System-552 pointing to this one.

The MPS approach to model generation is very similar to normal compilers. It should be possible to generate only a part of a project. In java you can compile only one source file, in MPS the smallest unit is a model.

If you generate one model, referenced models could be not generated yet. So you cannot rely on their output or intermediate models. Instead you should produce output given the very first existing input models.

There are many ways to achieve that goal in your own language. But the idea is still the same. The number of external nodes you need to generate a model is quite few. You probably don't need every existing model, but only ones you refer to. Besides, you don't need every node from that models except the "exported" or "public" ones. Talking about java, the only data you need about existing classes is their methods, fields and inner classifiers, i.e. an interface, not the implementation. The solution is to import this external APIs into our model in the beginning and process them along with the current model.

Let's consider an example. We have two models X and Y. The first one contains higher-level abstraction of classifier (like state machine). The later one refers to the first one.

Here is the reduction sequence:
X: ConceptA -> ConceptB -> Classifier -> java class
Y: ConceptARef -> ConceptBRef -> ClassifierType -> ref to class in .java file

If ConceptA and ConceptARef live in the same model, everything is clear. When they are separated there is a problem: we cannot refer to ConceptB from ConceptBRef. The target doesn't exist while we generate Y. It only appears in intermediate models of X. OK, we need to make this reference internal.

The obvious ways are:

1. Whenever you need to refer to ConceptB, copy it into the current transient model, as a root, but mark it somehow to delete at the last step (to avoid text generation). That is exactly what Java compiler does when you import external classes: it reads them (without method bodies, of course), stores their interface in the memory and probably processes them.

2. Create an internal reference concept, containing all required information about the target (method call can contain types for all parameters for example). For baseLanguage we created such internal concepts in generator-specific baseLanguageInternal language.

I've made several different attempts to implement 1 or 2 and am not having a lot of luck with either. They don't seem well supported in the system. Can we get a bit more detail on which mechanisms we should use to do this? Solution 1 sounds a little better to me than solution 2, but I've tried various attempts at both.

Ideally could you demonstrate using the simple example I posted in this thread the supported way(s) of doing this?

Something interesting happened. Shortly after I posted the message on Oct 30 my main project I'm working on started just working with simply doing by-string ref lookups and none of the other techniques you mentioned (it hadn't been working in the same configuration before). I recently got in to some strange (seemingly) unrelated state where I couldn't build anything so I deleted all my _gen directories and tried building from scratch and now my by-string references aren't hooking up again and I'm back to being blocked by this problem.

Can you think of any reason they would have been working for the past week or so?

With some help from Markus Voelter I got the simple example I posted working in some respect by importing external references. I haven't yet figured out how to then delete those references just before textgen, but at least this should be enough to unblock me.

The thing I was missing was I needed to add a separate mapping configuration for the preprocessing script that imports the externals that happens before the main mapping configuration.

I've attached it for anyone interested: CrossModuleReferenceTest-importExternals.zip (171KB)

I must grub out this thread since it's not clear to me how to reference to output nodes (java classes) in different models, especially accessories models. Can please some give a short description how to solve this, in my opinion, very common issue with the current MPS version (2.5).

At the moment I just use string reference macros. So I can succeed generation but java compilation fails because of missing references.

I've tried to copy imported root nodes to current model during generation. I'm not sure if this will solve my problem, but this attempt fails early because I can't see any of the imported models. I iterate over

model.rootsIncludingImported(GlobalScope.getInstance() , <no concept>)

in a pre-process mapping script, but I only see nodes from the current model and none from the imported ones.

Sorry, but I must annoy you with this thread... But this topic is very essential for my work with mps. And I don't see the sense of accessory model, if I can't use them in generator or reference their output. So please tell me, how you are using such models!

Is there again something broken with this forum? There are very few posts in here last weeks, especially from persons I suspect to be developer. Also the read count of the newest post is hardly increasing.
Is there any other way to ask question like the above?

Nothing is broken with the forum (at least to my knowledge). I regularly receive email notifications. The read count changes only when people go to the web page. It is possible for the people to receive notifications, read them and not visit the web page.

Thanks for your answer, Timur.
But as far as I understand this forum, you get notifications for threads you have posted something or marked as favorite. Ok, there is also a rss feed to read new threads, but in my rss reader the newest entry for this url http://forum.jetbrains.com/_rss/forum?forum=Meta-Programming-System is from the 9th of May. So there is seems to be some problem... perhaps just with my reader (Opera). If the important users here don't use the forum directly, its advanced features like voting and read count make not so much sense.

Since my problem has been discussed here before, I really can't imagine that no one can or want to give me a small hint.

I just recognized that I can get a IScope as well from the genContext and from the operationContext. I used them in the model.nodesIncludingImported() operation call to iterate over the nodes from imported models. But it's still the same behavior as with the GlobalScope before. There are no nodes from imported models.
What's wrong with my approach? Is there any documentation to the IScope and all this model/module dependencies stuff???

Models are generated one by one. During generation you cannot reference to output nodes of any other model except the current one. For Java we use two approaches:
1) import generated classes as JavaStubs. You can refer to them by qualified class names (for example from reference macro). Usually these classes are a part of runtime of your language.
2) use baseLanguageInternal, which allows to convert references to text as early as possible. The main disadvantage is that subsequent generator has no access to reference target, which often requires wrapping your expression into TypeHintExpression.

IScope provides a set of modules which are visible in the context (IModuleScope is a better name). In most cases you get scope for a module, i.e. it contains all modules visible (in terms of dependencies) from the current one.

GlobalScope/ProjectScope should be used only in dialogs allowing to add new dependency or to define scope for Find Usages tool.

Hello Evgeny,

Thank you very much for your answer! Both approaches seems to make lots of overhead for me. I just want to supply some common root nodes, which could be used and should be available if the accessories model is imported. And I don't want to create a special generator for imported roots.
So I guess copying imported models to my current model will be the best way for me. But therefor I must solve the problem with the nodesIncludingImported operation, I mentioned here. Do you have any idea how I can access imported nodes from within the generator?

But wouldn't it be a nice feature if you could access output nodes of an accessories model from your sandbox model. Because accessories models are part of the language they must be build before you create your normal models.

Importing all root nodes from visible models could end in performance problems. I suppose you invoke nodesIncludingImported on transient model (which doesn't have any imports). Try:

genContext.originalModel.nodesIncludingImported(<scope> , <no concept>);

Great, that's it!!
Many thanks! But this is not very intuitive.
I haven't thought that the input model for the first generator is already a transient one and also I don't know that imports are removed in transient models. There should be a warning if you use nodesIncludingImported with a normal model you get in every generator method.