Help with creating own StubCreator

Michael,

To load some external data into MPS models, you need 2 things:
1) a path from which the data is loaded
2) a "manager" that knows how to load a data from this path

So, you create your "manager" that can load stubs of you type (Excel stubs) from a given location. After it, you can add (Excel) stub models to any of your modules by opening "module properties" dialog and adding a new entry to "libraries" list as described in the last paragraph of the "Usage" part of the document you've mentioned:

Let's see how to add a new library of a new type to solution:
1. Go to Solution Properties -> External code -> Libraries
2. Add a new library (select a path where it is situated)
3. Choose an appropriate manager for this library>

---

What is a ModelDescriptor?

In MPS, there is a so-called "model descriptor" (inheritor of BaseStubModelDescriptor) for each model, which consists of:
1) model's properties like UID, name etc. that are not described in terms of MPS nodes
2) model (class SModel), which contains nodes
The SModel itself is loaded in lazy way (only when some nodes are requested from it), but the lightweight model descriptor exists all the time.

E.g. when you see a model in the Logical View, there' s no need in loading it's SModel, so only a ModelDescriptor exists at the time. But when the model is expanded in tree, we need to show its nodes, so MPS loads the SModel and shows its nodes in tree.

What do they do?

The "modelDescriptors" creates descriptors for models that can be loaded from the given paths. Then, those descriptors are registered by MPS and you can see those models everywhere (logical view, ctrl-n, etc.)
The "updateModel" method is used to load the SModel when it is requested by MPS.

What should I do in which method?

"modelDescriptors" - analyze the given location and create model descriptors for all future "models" that can be loaded from that location
"updateModel" - fill the given SModel with nodes ("load" it from the given location)

When are they executed?

The simple answer is "when the IDE decides they should be executed".
The longer variant is: _FOR NOW_ the "modelDescriptors" is executed on module load and after classes reloading, "updateModel" - when the model's nodes are required. In MPS 2.0 it will work in another way (though I think we'll be able leave the current interface of StubsCreator unchanged).

BTW, you can see some examples of Stub Creators by navigating to StubCreator concept (ctrl-n -> StubCreator OR ctrl-shift-s on an instance of StubCreator) and finding its instances in MPS (alt-f7, check "concept instances" and "global scope").

Regards,
Mihail

Hi,

I still got a problem concerning the Stubs.
I know some stuff about SnodeIds and ModelIds. (Is there some documentation?)

But I dont know when to use a DynamicReference and when a StaticReference.
In the ConfStubs both are used.

What I want to do:
I want to create a reference to another stub-model's node.

Hi,

It seems that there is no documentation on SNodeId &  SModelId for now. But it is quite obvious - e.g. SNodeId is a unique identifier of a node in scope of model. Actually, the ID can be anything, but we use 2 types of ids - one for nodes in regular MPS models (Regular) and one for stub models (Foreign). The second one is needed because it should be the same for the same node, regardless of when and under what consequences we load the containing model. This is because if we have a reference to this node, it should not be broken, for example, on a next MPS startup. For Regular ids there's no such problem, because they are stored as a part of a node inside of MPS model file.

About Dynamic and Static references - you'd better use a Static (as it is done in JavaStubs). Normally, when you need to reference some node, you should know its ID. So, a Static reference can be used. Dynamic references are actually references resolved by name at the time we ask them for a target node. ConfStubs is quite an advanced example, so maybe there Dynamic references are appropriate. I do not recommend using them in your own stubs.

Regards,
Mihail

Thanks for your answer. I already figured out most of the stuff on my own. But it's nice to see, that i'm doing it the correct way.

Although I still got some questions left:

There's SModelId and SModelFqName.
I already figured out that the FqName is used in the references to nodes in the model.
The SModelId is used to identify a Model.
Could I provide different versions of a Model with same FqName and different ModelId?

And I have a bit of a performance problem here:
My story:

I'm trying to import the data definitions (sql-like-tables with datatypes, domains and foreign-key references) to MPS.

My problem is that it's over 350 000 tables with overall 6 000 000 fields. (I'm not talking about table data, just the definitions).

What I'm trying to do now is to split this up to several Modells to improve the speed when starting the editor and pressing Ctrl Space.
(I tried it with 83 000 and it took 11 sec. to load the model but several minutes when pressing Ctrl Space.)

Is it possible to reduce the number of suggested items when pressing Ctrl+Space? Like when using Ctrl N.
Or could i provide some kind of custom search dialog that maybe is similar to Ctrl N?

The idea of splitting it up to several models is still a bit of a problem, as the tables foreign keys and domains are between those models. So if I would import only one of the models there may be some references that cannot be resolved.

Could I provide different versions of a Model with same FqName and different ModelId?

SModelId is a real identifier of a model used all over MPS. FqName is a visible name of a model, it's just a human-friendly string.
So, yes, MPS should be able to correctly handle models with same name (actually, that was the reason for introducing SModelId).
In some cases MPS can use model name (e.g. when trying to resolve a reference by name), but it's not a common case.

Is it possible to reduce the number of suggested items when pressing Ctrl+Space? Like when using Ctrl N.

I'm not sure I understood you right, but I'll try to answer. I see that you (or maybe we?) have a performance problem with completion menu.

So, first, the number of suggested items is reduced while you are typing, so in this aspect it works same to Ctrl-N dialog. If you've already typed some prefix, only those suggestions that start with the given prefix, are displayed. If you typed all the "name" of a field, and there are no other items that match that name in completion menu, you don't actually need to press ctrl-space.
Second, maybe the problem is not that you see too much fields in a table, but see all the fields in all tables in completion options? In this case, you can just create a "constraint" aspect for your "table field reference" and specify a scope for it. As you have about 20 fields per table, there shouldn't be any problem with completion menu then.

---

Actually, what MPS does when you try to invoke completion is like:
1) find all suitable objects to reference in scope
2) check constraints (can be child/parent)
3) displeay only those with the entered prefix

So, if you have a "scope" for your reference that works fast enough, not very complex constraints and not too much nodes in the scope, completion should work instantly.

Please do not hesitate to ask me for further details,
Mihail

Hi,

thanks a lot for your answer.

About the Ctrl N thing.
If you press Ctrl N and type an "A" there are like 1000 of possible Nodes. But in the list are only 50-200 displayed and a "..." entry at the end of that. So i think this would improve the performance, if i could say just provide 100 sugesstions and then stop and force the user to type another letter.

The problem is not the field-constraint. But i think 350 000 tables is already a problem.

Do you have an idea on the splitting up the models into smaller parts?

The idea of splitting it up to several models is still a bit of a problem, as the tables foreign keys and domains are between those models. So if I would import only one of the models there may be some references that cannot be resolved.

Is there some kind of lazy loading of depended models? So that the dependet Model is only loaded, when a node with a reference to another model is actually used.

In the StubsCreatorDeclaration (aka StubManager) there is an optional rootNodeDescriptor. Could that help me with my performance issues? When is it called/used?

Thanks in advance,
Michael

About the Ctrl N thing.
If you press Ctrl N and type an "A" there are like 1000 of possible Nodes. But in the list are only 50-200 displayed and a "..." entry at the end of that. So i think this would improve the performance, if i could say just provide 100 sugesstions and then stop and force the user to type another letter.

Thanks, Michael. We'll consider such a possibility. Now there is no way provided for doing such a thing in MPS, as far as I know.

The problem is not the field-constraint. But i think 350 000 tables is already a problem.

Ahhh... Now I got it.

Do you have an idea on the splitting up the models into smaller parts? <...>

I see no problem here. Let's consider a common case there. Suppose you have models MA and MB, and there is a reference from MA to MB. So, if you import MA into your model (called MM), you don't see objects from MB while editing MM. BUT there is no broken references in any of those models. If you really need MB, you just import it. MPS subsystems like typesystem will still see the reference from MA to MB as resolved (it doesn't depend on an "external" MM model).
The only thing you'll need to do there is the generator. For now, it should be possible to generate MB without generating MA. We had the same problem with baseLanguage and its classes - you can call a method of a class in MB from the MA model. We've resolved it by introducing the baseLanguageInternal wich allows us to reduce method reference to a textual "reference" and then just generate text from it.
We are resolving this issue, and (personally) I think there's no real problem, just a couple of things to be done. But you can still overrun this problem by using the same hack.

Is there some kind of lazy loading of depended models?

Yes. Models are loaded only when their contents are requested. Moreover, in MPS2.0 the first time model is loaded only to "root" level, which means only properties of root nodes will be loaded, not the inner nodes (they are loaded when requested). We plan also to split the model and store root nodes separately in future (think it would be implemented in 3.0), so only requested root nodes could be loaded.

In the StubsCreatorDeclaration (aka StubManager) there is an optional rootNodeDescriptor. Could that help me with my performance issues? When is it called/used?

It is called every time the user presses ctrl-n to show entities from stub models in list.

On completion feature - created issue:
MPS-12669

Thanks a lot, especially the thing with the references between models wasn't clear to me.

So what I could actually do is to split the models that are used only by dependencies (e.g. the data-type/domain-definitions used by the table-fields) into several small modells(like a model for all domains with prefix "AA", another "AB" etc).
And this would improve speed, cause the model is only loaded when a table is used, which field uses this domain.

GREAT!

But to solve my 350 000 table problem, i would need some other idea.
1. I would need a way to find the table without loading all the models. Hmm... thats impossible.

2. I would at least need some way to figure out in which model a table is. E.g. You want to use table "CustomersData" and the tool says: import "CRM-Model". Any ideas on that?

What is the scope of Ctrl-N? Imported Models? All stubs?
I cant find any root-node of my model. Do i need to provide a name to the models? (Currently it's only a getter.)

Could I probably create my own Ctrl+N?

1. I would need a way to find the table without loading all the models. Hmm... thats impossible.
2. I would at least need some way to figure out in which model a table is. E.g. You want to use table "CustomersData" and the tool says: import "CRM-Model". >Any ideas on that?

See ctrl-r (import model by root). It works the same speed as ctrl-n and allows to import a model by a name of a root node (table, in your case) inside this model. In MPS 2.0 it loads models only to root nodes level.
If you need roots from stubs, just implement getRootNodeDescriptors in your stub manager. If it takes too long, you can cache those values.

What is the scope of Ctrl-N? Imported Models? All stubs?

Project models. Or, if you checked the checkbox, all models seen to MPS.

I cant find any root-node of my model. Do i need to provide a name to the models? (Currently it's only a getter.)

Are you talking about regular MPS models or stub models? If stubs, then it's most likely that you haven't implemented getRootNodeDescriptors(), because that's how this works for stub models. If the model is regular, a property "name" is used in ctrl-n and ctrl-r dialogs

Could I probably create my own Ctrl+N?

Don't think so ;)

Tried implementing getRootNodeDescriptor...

But it needs to return set<StubDescriptor> and StubDescriptor needs an IClassPathItem and that needs a lot of strange stuff that is java-specific.
e.g. ClassifierKind, accept for the VisitorPattern, timestamps, subpackages, rootclases etc.

Michael,

It was our mistake because we haven't had enough use-cases. This is going to remain the same in 2.0, but I'm planning to change this in 2.1.

StubDescriptor should actually be an interface, not a class.
Anyway, you can inherit from StubDescriptor and just re-implement all its methods. When calling the super-constructor, you can just pass null for all its three parameters.

Not to forget, I've created MPS-12676

Thanks for your attention to MPS,
Mihail

Thanks a lot for your help.

I'm going to use sqllite as a step before the stubloading takes place to handle my several csv-files and i hope that will increase performance as well. (for the loading process at least).

This is a good solution for that moment.
Later, most likely after 2.1, we plan to implement stub models caching so that long loading will be a lesser problem. I mean, if a stub model parsing is long, we can cache the model in a binary format and only synchronize it with source when the source is changed.
Regards,
Mihail

I have another question about the ModelDescriptor-Stuff.

First I'm still a bit confused about the SModelID and the SModelFwName.

Second (more important) the SModelFqName has a fqname and a longname. Is the longname only used for presentation? Can i provide somekind of description here (e.g. how many items in that model)?

And what is the stereotype? Should it be the same for all the models that are loaded using 1 StubLoader? Or can i use different names here?

Another point:

The search behaviour of "model import" (e.g. Ctrl+M) is really strange. I type the first part of my model-name and then it vanishes...

Why isn't it like a fulltext-search?

Does it have problems with uppercase character?

Is the longname only used for presentation?

Mostly yes. In some rare cases (when no module id is present) MPS can use it for module resolution, as far as I know. In addition, it is now used when generating folders for resulting text files.

Can i provide somekind of description here (e.g. how many items in that model)?

I wouldn't recommend doing this. Here in MPS, we think of models as of Java packages, so the name of the model is just a name of a package and can be used where we need it (the only case where model id is used is references resolving), so I can't guarantee this will work (even if it works at the moment). But, anyway, you can inherit from SModelFqName and create a new field in inheriting class which can contain some description. If you need some other features, which we don't have (e.g. customizable showing of such model names in logical view etc.), just ask us - and we'll try to implement. Or you can do them yourself and contribute into MPS.

And what is the stereotype?

Stereotype is an additional info about meta-type of the model. It is used for defining specific handling of some models. For now, we use a number of them:
EMPTY - a regular model
@generator - says that this model contains generator templates, that should not be generated theirselves
@SOME_stub - stub model. Find usages, ctrl-n and some other subsystems of MPS treat those models separately from regular models.

Should it be the same for all the models that are loaded using 1 StubLoader? Or can i use different names here?

No, they can be different. But all of them should be in "<XXX>_stub" format. This is because we have SModelStereotype.isStubModelStereotype() which determines whether the given model is stub or not by the suffix of its stereotype. It's not very discoverable now, but I think we'll change the stubs language accordingly in future, so this will not be a problem.

Michael,
What version of MPS do you use?
There's a bug in current MPS version with ctrl-m, you just need to type a  "short" name (after the last ".") to see your model there.

Why isn't it like a fulltext-search?

Actually, it is supposed to work like ctrl-n works with nodes and ctrl-n in Idea works with classes.
So, you can enter either a short name of your model or start with a prefix like "jetbrains.mps." or just "j.m.". The string you enter is converted into some regexp and then names of all models are tested with it. Matching are displayed in list. We found it convinient if ctrl-m works this way.

Does it have problems with uppercase character?

It is case-sensitive. Furthermore, it supports CamelHumps and pattern matching. I don't actually know about any problems there, but if you have found some, just report, and we'll fix.

StubDescriptor should actually be an interface, not a class.
Anyway, you can inherit from StubDescriptor and just re-implement all its methods. When calling the super-constructor, you can just pass null for all its three parameters.

Hi,
sadly it doesn't work as i expected:
I got my concepts into Ctrl+N, but when clicking on them i get this error message:
Can't find model descriptor for: f:java_stub#44ac43-33ff-4377ff-543f-cedd#LO(de.michaelbrunner.sqllanguage.sandbox/LO@java_stub.

What do I need to return in getClass or getPackage or getConceptFqName to get this work?
Or is it not going to work because of the java_stub things?

Yes, it's because of this @java_stub. Thanks a lot, this is a serious problem.

Michael, are you using the 1.5 version, M5 or a raw repository version? I will fix this immediately, and will rewrite the StubDescriptor as an interface, so you can try it in M6, or get a current repository version, or I can just make a build from latest repository version and share it with you.

Thanks for report,
Mikhail

Michael,
I've fixed this. Now there's an interface IStubNodeDescriptor:

public interface IStubRootNodeDescriptor {
  String getName();
  SNode getConcept();
  SModelReference getModelReference();
}

If you have some suggestions about this, you can write them here.

Is it really an SNode or is it an SNodeID?
Ah, I understand, with SNode you can show the correct icon.
Could you show the shortDescription, too?

I'm using M5. Do you know when M6 is going to come?
If you want me to test this feature before M6 I could do this.

Other thing:
As mentioned above, I would like to show some kind of description about a model to the user, that isn't used for identifying the model.

So my model's name is AS and the description is "Automatic System for Car Insurance".

Do you want me to make a ticket for this?

Could you show the shortDescription, too?

I think, yes. The string shown on the right side of the ctrl-n list is provided by us, so we can give developers a way of changing it.

I'm using M5. Do you know when M6 is going to come?

We think about building MPS 2.0 RC1 in about a week. I don't know when it will be published, but think it will be nearly the same time.

If you want me to test this feature before M6 I could do this.

Thanks a lot ;). If you want to try it, you can. But as you use M-builds, it may be more convenient for you to wait for M6.

As mentioned above, I would like to show some kind of description about a model to the user, that isn't used for identifying the model.<...> Do you want me to make a ticket for this?

Michael, could you please describe where you want this description to be shown? Ctrl-N list only? Model properties dialog? Some other places? If you do this, I could create an issue just with a link to your message; or fix it immediately if I have some time.

Actually I would like to show it at all the places.

Most important is Ctrl+N.
Then also when editing the Model-Imports.

In the logical-view when browsing the stubs is just a nice2have feature.

Filed: MPS-12769
I'll talk to the team and, I think, we'll implement this feature.

Thank you for the idea.

Hi Mihail,

I've tested it and it is working good!
I can now import models using Ctrl+R!

Great!!!

Thanks a lot!