importing jars: ASMClassType "out of search scope"

snapshot of 2019.1 importing docx4j 8.1.3

snapshot of 2019.2 importing docx4j 11.1.3

You can try 'Show Node Info' action (Ctrl-Q) on ASMClassType node to find out detailed reference information. You'll see the model/module name there to help you understand where MPS tried to find the node. 

In MPS 2019.2 (as well as in 19.1 and 19.3), it was MPS.Core solution to expose ASMClassType et al. In 2020.1, these stubs are part of jetbrains.mps.java.stub solution. However, given no change between 19.1 and 19.2, I suspect ASMClassType was exposed by some other solution, perhaps it was old version of docx4j that exposed these? FWIW, I would refrain from using asm classes through MPS.Core (as you'd need to change it once again when switching to 20.1). Instead, I'd expose asm java stubs the same way you expose docx jars.

Thanks for the quick reply. Node Info says it is looking for `jetbrains.mps.baseLanguage.javastub.asm` but I cannot find such a module in my module pool.

I added `MPS.core` to the dependencies too (keeping JDK and jetbrains.mps.java.stub) but the problem persists.

What do you mean by "exposing asm java stubs the same way you expose docx jars"? I tried very bluntly to copy over "asm-all-7.0.1.jar" from MPS lib to my solution and put it on the source path just like the docx4j jars, but that did not solve it. That jar seems not to contain the ASMClassType class.

I have located the ASMClassType in the module pool, but I am not able to create dependency to it from the Solution Properties nor find a jar that seems to contain it. What am I missing?

There's a bug in MPS in the part of the code that processes annotations, they all get imported with this ASMClassType class even though the actual sources refer to a different class.

Here's a portion of the decompiled sources of the P class:

package org.docx4j.wml;

...

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(
    name = "",
    propOrder = {"pPr", "content"}
)
@XmlRootElement(
    name = "p"
)
public class P implements Child, ContentAccessor {
    private static Logger log = LoggerFactory.getLogger(P.class);
    protected PPr pPr;
    @XmlElementRefs({@XmlElementRef(
    name = "customXmlMoveFromRangeEnd",
    namespace = "http://schemas.openxmlformats.org/wordprocessingml/2006/main",
    type = JAXBElement.class
), @XmlElementRef(
    name = "ins",
    namespace = "http://schemas.openxmlformats.org/wordprocessingml/2006/main",
    type = RunIns.class
), @XmlElementRef(
    name = "customXmlInsRangeEnd",
    namespace = "http://schemas.openxmlformats.org/wordprocessingml/2006/main",
    type = JAXBElement.class
), @XmlElementRef(
    name = "subDoc",
    namespace = "http://schemas.openxmlformats.org/wordprocessingml/2006/main",
    type = JAXBElement.class
), @XmlElementRef(
    name = "moveFromRangeStart",
    namespace = "http://schemas.openxmlformats.org/wordprocessingml/2006/main",
    type = JAXBElement.class
...

It doesn't refer to ASMClassType anywhere.

Indeed so. Well spotted Sergej, thank you. I checked with the docx4j source on github where it says (see below)

So what's next? Should I file a bug report? Will JetBrains do that for me? Please advise.

This leaves me puzzled how come this worked in 2019.1 with docx4j 8.1.3 ... and If I could get away with this bug in 2019.2 too -- provided I can get the ASMClassType resolved. This could keep me going while the bug is being fixed.

Guess, it's https://youtrack.jetbrains.com/issue/MPS-23087. 

No idea how come it used to work with 2019.1.

Does the error prevents you from moving on?

Meanwhile I tried a brute force workaround. I figured out that ASMClassType is contained in mps-persistence.jar. I put this jar on the source path of the solution and as a result ASMClassType is part of the java_stubs in the solution now. However, it still does not get resolved in the docx4j classes. So this  is not a viable workaround.

The error prevents me to migrate from 2019.1 to 2019.2 which I need to have completed before the end of March. In our company all language developers migrate at the same time. So yes, in 4 weeks from now this bug will be blocking -- assuming we do not find a workaround -- for which I see no further options because the jars are a given, there is no way to tweak them.

One more thing: we are always 1 release behind the latest MPS release, because we work with a distribution managed by itemis. This distribution contains many extensions and most of mbeddr, iets3 etc. and a lot of command-line build stuff. The latency in versions is the time it takes itemis to stabilize the distribution (this time it took much longer because of the Java 8 to 11 transition). So if you would fix this bug and backport it to 2019.2 that would be awesome. If you only fix it in 2020.1 it will take half a year before the fix reaches me ...

Can't you just ignore the bug?

Not sure yet. In 2019.2 the generator stopped working because of JAXB issues (which I did not have in 2019.1). So it seems that the bug of having the _wrong_ class in the annotation does not hurt, but the bug of having the wrong class _unresolved_  does hurt. I would have to investigate. 

My problem is that up till now I could treat the docx4j jar set as a perfect black box, focusing on building a language that mirrors the WordProcessingML standard - a challenge by itself. This bug makes me go inside the black box.

To find out whether this annotation is the only issue that prevents you from moving forward, try next workaround:

1. Add jetbrains.mps.java.stub solution to module dependencies of a module that holds your java stubs

2. Go to console window, type next code and then execute it (well, you'll need to use code completion for certain parts like model names, you can't just type their names in):

new ModelImports(model-ptr/org.docx4j.wml@java_stub/.resolve(#project.getRepository())).addModelImport(model-ptr/jetbrains.mps.baseLanguage.javastub.asm/)

(you'll need to import ModelImports class, hit Cmd-R twice to access stubs, it's in MPS.Core/jetbrains.mps.smodel@java_stub)

ModelImports argument is the model you'd like to fix imports for, I guessed it's org.docx4j.wml@java_stub in your case (the one that holds P class). If there are more models that need to be fixed, you'd need to execute the code for each of them. Argument for addModelImport is a model reference to the model that holds ASMClassType. With that, reference t ASMClassType from annotation could be resolved (you need to re-open editor if it was open to see the difference). Then, try the rest of your process to see if this broken reference was the only obstacle.

Don't try to add imports to the model through Model properties dialog, java stub models are read-only from UI perspective and your changes won't get saved. ModelImports just do the same but ignores the fact model claim to be read-only.

Note, the change doesn't persist, the moment java stub model is reloaded, import is gone.

I performed the above step, and indeed, the ASMClassType references get resolved.

The problem is that there are >50 stub models that need to be processed (not all of them refer to ASMClassType, but I cannot tell unless I open them). Hardly feasible to process them all when doing this manually. So let's think of an SModel query to help here. How to iterate over all models in a solution from the console?

I tried to write such a query myself but failed. Help appreciated!

Console, with smodel, closures, baselang and other languages has power to do almost anything you need, it's definitely possible to iterate any number of models. You don't need to fit everything into single statement, start with a Block statement and write a program of whatever complexity.

SModule.getModels() gives all models within a module. To get a module, I'd go with module-reference// resolved the same way we did for model (#project.getRepository). Then, with getModels() you get SModel instance right away (don't need to resolve them any further). If it's only java stub models in the module, then you can proceed with ModelImports, if there are other models as well, you'd need to filter out java stubs (with SModelStereotype.isStubModel(SModel)). 

Thanks for the hints. Much appreciated. Console is new territory for me but I am eager to learn this.

This is how far I got (I do not know how to copy the console to the clipboard as text)

Here getModels() returns Iterable<SModel>. I have difficulty iterating over it.

The only iteration construct available from the completion menu seems forEach(<Consumer<? super @Iterable.T>):void. Although I have no clue what "? super SModel" means (and nothing to find in the MPS docs), I continued by writing a closure literal with parameter type SModel. Then this balloon appears. `error: type ? super SModel is not a subtype of jetbrains.mps.smodel.SModel`. I do not know how to proceed.

As an alternative, I imported language jetbrains.mps.lang.smodel.query to see if I could write a smodel query expression over the Iterable<SModel> but that did not work out as none of the smodel query items appeared in the completion menu.

So my question is: how to iterate over the Iterable<SModel>?

? extends SModel is regular Java generics, Iterable is just a regular java.util.Iterable you can iterate with any Java loop. The only thing to be careful is to use openapi.SModel, not smodel.SModel (there's unfortunate name clash).

{ 
  SModule module = module-reference/name.of.your.module.chosen.from.codecompletion/.resolve(#project.getRepository()); 
  Iterable<SModel> models = module.getModels(); 
  foreach m in models {
    if (!SModelStereotype.isStubModel(m)) {
      continue;
    } 
    new ModelImports(m).addModelImport(...); 
  } 
}

You can also use sequence<SModel> instead of Iterable<SModel> (sequence<SModel> models = module.getModels();) that would give you access to collections language constructs (select, forEach, where, etc). But for the task at hand, Iterable + for would be pretty sufficient.

Thanks again. Learning a lot today. I got to this query

To make the parameter of the closure literal of the interface SModel, I did: type CTRL+R, type “SModel”, choose the interface.

Now I will continue to investigate if the resolution of ASMClassType was the only hurdle. Maybe there are more. Stay tuned.

in the console I ran  ``#showBrokenRefs project`` (lucky find - I was tinkering with the console). I will now investigate the cause of the broken refs and if they are a problem or not.

The good news is: the broken refs are caused by the binary distribution from docx4j.org being incomplete (jars missing). It is not caused by the MPS javastub logic.

The bad news is: I tried to run the samples from docx4j.org. These are distributed as sources. I created a solution org.docx4j.samples with

and on the Commons tab point to the java sources and mark them as source. After compilation none of the examples will run. There are two errors all over the place: 1/ language jetbrains.mps.baselanguage.javadoc is not imported 2/ Unneeded java imports present.

Interestingly, actually I had included j.m.b.javadoc in the dependencies. 

But we may ignore this problem. I'd rather have the sample sources available in my project not as stubs but as baselanguage classes. I will do so so tomorrow using "create model from Java Sources".

To figure out if there are more blocking issues with importing jars in 2019.2, I did the following.

First I tried to check the models. There are 587 of them, doing this manually is hardly feasible. I could not find a way to run a checker on all of them (check module does not recurse into the models). Q: can it be done using the console? A random check revealed plenty of errors like "type parameter is not within its bounds" or "node '{' cannot be child of node 'value'". But there were too many of them, and I looked for another approach. What if I could just ignore the errors -- like Sergej suggested?

So next I created a solution for the 80+ samples provided by the docx4j.org project, and imported them using "New Models from Java Sources". This posed multiple problems, but that is a different story -- I will report elsewhere. In the end I could get all of them to compile. Then I ran some 10 of the samples, the ones that touch the functionality from the docx4j jars that I will be using soon. All of them ran without issues. 

So my verdict is: there are no more blocking issues with importing the jars. Once the ASMClassType references are resolved, I can move on.

So my next question is: is there a hook like "SolutionFinishedLoading" to which I can attach the script that fixes the references? Right now I have to run it manually from the console every time the project (re)loads, which is sort of acceptable during development, but it is not for deployment to other users.

Indeed, check models doesn't nest into 'submodules' (there are technically no 'submodels' in MPS, just models sharing same namespace are visualized as 'nested'). Check Solution action shall check all the models as well. 

There's no mechanism to hook for MPS events other than writing custom MPS plugin, which is quite a topic. With the MPS-23087 fix backported to 19.3, I'd rather switch to 19.3 than writing a custom code hook for 19.2.

I agree. Doing a plugin as a temporary workaround is hardly worth it. I am looking forward to a 2019.3 backport. I will check the 2020.1 EAP3 as soon as it appears.

I think this thread can be closed now -- thanks for your quick replies. Just one more thing. I want to keep investigating issues with jar imports -- in case I might be hit by one when expanding the mps-docxgen project. Is there a way to use the console to query java stubs for errors (any kind, or a specific one)?

This thread got me working with the console for the first time and I would like to learn to use more of its potential.

I propose I turn my console questions into a separate forum topic.

PostScriptum: I just figured out that 'Check Solution' normally iterates over all models of the solution and does a 'Check Model' on each, but this is disabled by default for stub models. To enable it go to Settings|Preferences > Tools > Model Checker > Check stub models. Running 'Check Solution' on org.docx4j reveals there are quite some issues when importing docx4j jars in 2019.2 (in 2019.1 there were zero issues). I assume there are many duplicate issues, I will remove duplicates and report back, presumably in the MPS issue tracker.