How to organize large projects in MPS?
Hello everybody,
we are looking for some best practice/guidelines for storing and organizing large projects in MPS.
We built a DSL as replacement for XML-based input sheets. We also wrote an importer to transform/load existing XML files into the MPS model. Reusing existing data is very important.
But we realized that managing large data in MPS IDE does not seem trivial. We would like to transform something between 1000 and 3000 XML files (each will be a root node in MPS). If we store everything in one model, the model will either be stored in a file of around 100 to 300 mbyte or in 1000 to 3000 files. Both solutions slow down the IDE significantly.
We would like to ask for some best practice / experiences / guidelines for storing and organizing large projects?
Should we split the data into several models, or solutions, or even projects? Some files depend on other files, but it is not necessary to have everything always loaded.
Thanks in advance!
Best
Sven
Please sign in to leave a comment.
I would avoid single model approach (either single-file, or per-root persistence) at all costs. Model is a unit MPS often operates on as a "complete entity", e.g. for purposes of m2m/m2t transformations, to load/save, to find nodes, etc. MPS does support some limited optimization not to load complete model the moment modules/models get discovered, but once any code accesses anything but properties of a root node (e.g. navigating SReference or accessing a first-level children), complete model would need to get loaded (and it doesn't help if it's per-root persistence). Indeed, it's an option to have your own model persistence that could be optimised not to load everything (e.g. a DBMS-backed), but it's an endeavour of its own.
I'd say breaking clusters of related XML data into modules, and then further down into individual models is the way to go. 3000 models (one per XML) is something MPS could definitely handle (there are 6000+ models in MPS itself, and much more if there are extensions, like mbeddr, installed).
Hi Sven,
MPS absolutely hates large models but it can handle tons of small models really well. That is also the reason why the JDK isn't "imported" into a single model but into one model per java package.I would suggest looking into a way to create small but many models from your data and make the importer smart enough to set the model dependencies correctly. MPS has some tricks behind the scenes to only load the really required models completely and will only parse the meta data header for other models. Essentially lazy loading them as they are required.
That said you might still run into problems in scenarios where all models are loaded e.g. during generation in the CI system where you will need to keep an eye on giving the VM enough heap space.
In my experience Module aka Solutions/Languages don't have much of an effect on performance in general, having one Solution with lots models isn't really a problem.
A a rule of thumb: what is typically a single file should be a model in MPS.
I know base language doesn't follow this rule and it's not always 100% that easy but its often helpful to think of a model in similar way as file and root nodes are then an additional layer for organization below.
Thank you very much Tikhomirov and Kolja :) This looks very helpful.
The files are already organised in a folder structure and instead of using virtual folders we will create multiple solutions and models.