Building a DSLs-Zoo realized with Mps

Yes. :)
Perhaps I could take this as an occasion to write some more docs in my public MPS examples on github and translate screencasts into English...

That's a great idea. The time seems ripe to grow a community around this technology. Perhaps a slight redesign of the MPS pages could accomodate a DSL-zoo? When possible, e.g., open-source projects, the zoo should link to the MPS plugin repo (http://plugins.jetbrains.com/plugin/7112?pr=mps).

Beyond this, what I would find the most useful to build a community is an additional way to offer documentation for MPS. The user guide is useful, but often not updated when new features adjust the behavior of an old version. The tutorials are good when they are up to date, but they often fall out of sync quickly. In this case, I would suggest adding a short 'feature tutorial' whenever a new feature is developed. Each feature tutorial would be short, rather than a full length tutorial driven by an example, and explain concisely how to take advantage of the new feature. The page would include the version of MPS for which the tutorial was developed so that iterations on the feature yield different pages that help users see differences between software versions. Having a way for the community to contribute feature tutorials and offer them in the same place would be a good way to grow documentation (perhaps a git repo?).

Hi Jens, Hi Fabien,

I am thinking both about documentation (tutorial, examples, screencasts) and about sharing common experience. For the latter point I think the most interesting part would be to be able to share whole projects and discuss them altogether.

This is particularly throw for implementation of languages or subset (e.g., JSON for MPS) which could be shared instead of be re-written over and over, and useful extensions to Java (like the memoization extension showed in a tutorial from Jetbrains). Other projects would be very useful to show how complex stuff can be implemented in MPS. See for example mbeddr (mbeddr.com).

We should think of a way to build this community that is easily accessible (for example the plugins list is not: I wrote to they guys to add a small DSL I developed and I got no answer), something like a wiki for example, but also something with encourage people to contribute. Someone could prefer to post the material he/she produces on its own blog or website, so this community could act as an index or a collector of stuff, maybe.

Another limit of the plugins list is that there is not so much "metadata": there is not a nice description, tutorials, example, screencasts, ways to contact the authors and what about the source code?

I wrote to they guys to add a small DSL I developed and I got no answer

Have you tried to upload your plugin (click Add Plugin in your profile)?

something like a wiki for example

The wiki is located at http://confluence.jetbrains.com/display/MPSD25/MPS+User%27s+Guide. It's confluence based (I've only experience with media wiki) and until now I've some problems using it - e.g. I did'n find the button to edit pages... ;)

This is something I have been wanting to do for a long time, and I'm excited to see others are interested in the same thing (sometimes I wonder if I'm the only one still using MPS, since the forums can go dead for days on end).

I don't want to step on any toes, but unless someone else is already doing it, I don't mind stepping up and getting a website up, cleaning up and contributing some of my languages, and working with others to integrate theirs into a useful core of languages to serve as a jumping off point.

I've been thinking for some time about how to build up more of a community for MPS, and a language zoo in particular, and here are a few things I think are worth considering:

1) Documentation is important, of course.  About 6 months ago, I prototyped a simple language-documentation language that allows me to attach comments at the concept level.  I originally did it as an exercise in using annotations, but I think with a couple of enhancements it could be very useful for this community effort.  First, I could add additional documentation types for children, relationships, and properties of concepts, as well as other language definition aspects.  Second, I think an IDE plugin that creates a simple tool window for displaying the documentation of the currently edited concept, allows you to browse the documentation, etc. would be very useful.

2) My personal view is that there are 2 broad categories of languages needed for using MPS.  At the highest level are DSLs for various business domains.  These will tend to be quite customized for various businesses or domains, and depending on the pervasiveness of the domain, may or may not be suitable for sharing or a community effort around building them.

The other category are what I think of as technical DSLs - all the lower-level stuff that your main domain languages need to build upon to be usable - MPS baseLanguage being the classic example.  This is the area where a vibrant community would not only be very useful, but I would argue essential for MPS to survive and increase in adoption.

Today, if you are doing strictly Java development, MPS gives you great support for your DSLs and you can ramp up quickly.  Anything else though, and there is a lot of work to do just to get the underlying support needed.  I believe most of these technical areas are more widespread - to use a recent example, I needed a JSON language, and although it only took me about half a day to write it, if 100 other developers have the same need, that's 49.5 days of effort wasted.  However, if we pool our resources as a community, then that time can be better invested by say, extending that JSON language to use custom persistence, so the models are saved on disk as JSON.

I would further divide the technical DSLs into 2 subcategories - base languages (i.e. the ultimate targets of generation - Java, JSON, XML, HTML, CSS, build scripts, etc.) and ??? - I haven't thought of a good name for the other.  But this category would be all those generally useful languages across many domains, such as entities, expressions, ui definition, access control, modules, components - all the "good" stuff.

3) I think a few best practices and guidelines agreed upon by the community would be useful to prevent fragmentation and promote maximum reusability.  But that shouldn't be a hindrance - if someone has a language that doesn't meet the guidelines, they should still contribute it, and the community can always refactor if needed.

No, I did not notice this possibility :)
On which page?

http://confluence.jetbrains.com/display/MPSD25/MPS+User%27s+Guide

I am thinking about something different from a User Guide, more like a collection of tricks and practices learnt while using MPS to build different systems. Right now it seems the only significant experience comes from the guys at Jetbrains. I am sure it is not like that: I am participating in the mbeddr project which involves many developers using MPS and I am confident there are other industrial usages of MPS. We just don't know about them.

Probably most of the user are facing and solving independently the same problems, this is something to correct, IMHO.

You can use your forum account to login (upper right corner on the plugin page). Then click on your name (upper right corner again) to go to your profile. There you can find an Add Plugin link.

This is something I have been wanting to do for a long time, and I'm excited to see others are interested in the same thing (sometimes I wonder if I'm the only one still using MPS, since the forums can go dead for days on end).

Good to know. I had also the feeling of being the only user (I started taking a look at MPS many years ago) and I wrote (with another account) a number of times into this forum. Until I took part in the mbeddr project I was wondering if anyone was really using MPS. I had the proof this is the case and I started wondering how many other users there are out there.

Myself, playing with MPS alone I did not manage to do that much, but having seen what the guys of the mbeddr project did 1) I got the feeling you can use MPS to build something real 2) I learnt a lot of tricks & practices. We should do the same, on a larger scale and sharing experience coming from different domains.

I don't want to step on any toes, but unless someone else is already doing it, I don't mind stepping up and getting a website up, cleaning up and contributing some of my languages, and working with others to integrate theirs into a useful core of languages to serve as a jumping off point.

Good to hear! What platforms where you thinking about? A wiki? A blog? A custom website?

I am interested in the stuff about documentation. There concerns which could be factorized and used in many sitauations (requirements is another one coming in my mind).

Maybe we could sharing our experiences (small languages developed, writing some tutorials) and then build something small trying to use all the best practices and describing the things learnt. I think we have a decent documentation on this website, what new developers (and people trying to use MPS in practice) miss is some "real project" from which to learn.

Of course what we implement to build this small showcase project could be also a base of components re-usable independently (for example the small JSON language).

About best practices, of course everyone would have different ideas, we need a community to confront and sometimes converge on the best ideas.

Well, as luck would have it, I have almost 2 weeks right now with absolutely nothing to do (in between contracts, new contract starting 7/8).  It looks like this is the perfect time to get the ball rolling on this!

For starters, I think the path of least resistance is just to use Github.  Besides the code repo itself, we can use Github Pages for the website and for now use their wiki and issue tracker for documentation and discussions.  Ultimately, I would like to have a full-blown community with blog, wiki, screencasts, etc.  But for the sake of getting the most traction while I have the most free time, Github is probably the way to go.

I've been thinking that a good starting point might be the documentation language and a generator for Markdown that generates the docs for the site from the code repo.  This gives us a practical starting point and a base we can use for all future language documentation.  Ultimately, I want to have a full suite of web languages (among others), and have done some work in that area.  So as we add languages and generators for HTML, Javascript, etc., we can use those to move towards a more full-featured website with blog, wiki, etc.

Federico, Jens, and anyone else interested in contributing, go ahead and contact me at mike at haneysoft dot com, and we can discuss more details.

For now, I am going to create a Github organization - probably will just use 'mpszoo' for the name, we can always rename it later.

One other thing we need to decide - which version of MPS?  With 3.0 being so close to shipping, I'm really tempted to say we should just start with it.  Normally I would say the opposite - stick with 2.5.x for stability, but there appear to be some substantial internal changes with 3.0, so I'm reluctant to saddle our new project with supporting an obsolete architecture.  Thoughts?

Hi guys,

this is a great initiative.
Some time ago, we were thinking in the team that we could build a wiki-based community page hosted by JetBrains, in the same way the IntelliJ IDEA project did - http://www.jetbrains.org
The infrastructure is not ready ATM, but I could check what can be done, if this would be your preferred option to using GitHub. Alternatively, we could use the MPS space at confluence, at least for some time - http://confluence.jetbrains.com/display/MPS/Welcome+to+JetBrains+MPS+Space

In any case, I do not want to hold you back. We can always migrate the site when the need comes.

As for the MPS version, I'd vote for 3.0 at this point.

Vaclav

You're right, it's headlined as User Guide, but technically its a wiki and the content contains links to other languages, tutorials, cookbooks etc - and there are some pearls hidden in the Child Pages section too. It's more than a plain User Guide. I thought that would be the right place to add our sceencasts, blogs, languages and so on as well as fix bugs in the existing documentation. Maybe Vaclavs page http://confluence.jetbrains.com/display/MPS/Welcome+to+JetBrains+MPS+Space is the better place as a starting point. Or is there a good reason not to use JetBrains' infrastructure? (Besides getting access rights to add and modify things, but that would also be true for any other place.)
And yes, I agree fully that it's time to share our tips, trick, knowledge, and languages on a common place. On github I've some MPS demos and I'm sure I could extract some languages from our 'industrial usages' for common reuse.

I agree with Vaclav that we could start on GitHub and later move to some JetBrains hosted resource.

Let's start discussing some initial contents that we could contribute.

Maybe I could implement a Json editor from scratch documenting each step (and we could later refine).

What else could we provide?

Maybe we could list existing projects and present them on this "community website".

Currently I have on GitHub:

• Demo for loading a file, parsing it's content and creating a solution model of it.
  
• Game of Life with baseLanguage extensions (including a decision table and a mapping table)
  
• MPS demo project for showing language evolutions scripts and custom menus.
  
• MPS demo DSL for data definition language
  
• a demo how to create a refactoring in MPS based on the constants language by Vaclav Pech.

Somewhere I've also the extension of the Money sample with attributed types so that only money of the same currency can be added. And as I said I have some common languages which can be extracted from projects like Gherkin and some WebService stuff. Maybe also some XSLT and FO stuff.
Nearly all has to be upgraded to MPS 3.0, but with the last version the JetBrains folks introduces a converter from 2.5 so that should be done faster than my first attempt with manual work...
Unfortunately my screencasts are in German only because I have to practice my English first...
And for the calender there is a talk at the Clean Code Developers Days in Dresden about why you should consider using a language workbench for clean code development (by me and also in German because the audience is german-only).

I agree that there are pearls hidden in the Child Pages, sometimes the information is there, just a little difficult to find :)

The Vaclavs page seem to contain the kind of stuff I was thinking for the community website, I am not sure a wiki is the only resource we would need.

I would suggest to have also a blog, if we could have 4-5 authors we could manage to have a post per week (it is circa one post per month for each author), this could motivate us to provide new material for the community and keep it alive. Having an alive community could also motivate others into participating.

A blog is a good idea. In our in-company blog I've written some articles that would perfectly fit in here. I think in average I could deliver a post per month.

I created the Github organization and added all those involved in the discussion to the group.  It's late here, but after I get up tomorrow I will start setting up the wiki, issue tracker, and a landing page.

In addition to the standard links to screencasts, languages, etc., I have an idea for content I want to bounce off you guys.  For a recent project, I dove into the world of single-page webapps.  This is an area of technology that is growing quickly, and experiencing a lot of growing pains, and I think an area where MPS could be used to great effect (I won't take up space here listing all the issues and potential MPS solutions - maybe that will be my first wiki entry for the new site).

Because of the overwhelming number of client-side MVC frameworks that have sprung up over the last few years, a project was created called TODOMVC to help people to choose.  They standardize the functionality for a simple (but non trivial) todo-list application, and provide shared assets (stylesheets, images, etc.) to ensure a consistent look and feel across implementations.  Each MVC project implements the application using their framework and submits it to the TODOMVC repository.  It's very popular, because most developers would rather look at real code than a list of "feature" bullet points when evaluating a framework.

So, my proposal is, we implement a series of DSLs to generate this TODO app from MPS, and we document each step of the way in a tutorial-like fashion through articles and/or screencasts.  I know it's a big job, but in the end we would have not only an excellent learning resource but also a set of real, useful languages everyone can use.

So you recommend code casts like Uncle Bob and Corey Haines on http://cleancoders.com/ ?
Web frameworks are surely a good field for DSLs. There is a huge amount of code for so little content, and there are already several languages like HTML, JavaScript, CSS and so on where IDEA does a great job for a 'text editor', but is limited by the text nature. I've prototyped a JSF based language for web development and saw how much cleaner one could code with a language workbench.

I really like this idea, also because I see language workbenches as the perfect place where to do language integration and web development require to mix different languages.

I think we could implement it step by step in a didactic way, discussing and documenting design decisions, maybe as blog posts, what do you think?

The "discussing and documenting design decisions" is the most interesting part. I read several sources (incl. mbeddr) but always I have to guess about the decisions. Perhaps we could also add a code review corner to our community portal. Or dating for remote pair programming sessions via skype and teamviewer.

It is great to see enthusiasm to increase community awareness of MPS. I also vote for MPS 3.

We've created a couple of simple MPS languages that we plan to release as open source in the near future. We'd be happy to contribute them to the MpsZoo. Here are a few of these simple languages:

• GString offers a groovy-like string in baselanguage, where you can embed references to varibles: "literal${a}2" will evaluate to the string "literalA2" if the variable string a was defined the value 2. We have intentions to turn multiple references written as string in a literal into string variable declarations.
  
  
• PathPatterns make it possible to expand file wildcards into filenames. For instance "*.xml" will expand to all finames matching *.xml in some directory.
  
  
• NYoSh (Not Your ordinary Shell) is our attempt to replace a shell with baselanguage extensions. It supports executing commands in a similar fashion to standard shells (e.g., running commands separated with |,  &&, ;, |& or ||). You can get the standard output of a command in Java as a stream or reader and process it.

These three languages are designed to work together. They provide key features needed to support command line execution script development with MPS (having code completion and a debugger when developing scripts is really nice).


Regarding content: when learning some new tool or language, a useful resource offers complete examples (the complete language you refer to), but also explains specific parts of the system with tutorials. If you look at the goby project we built over the years, see http://goby.campagnelab.org/, you will see reference documentation and specific tutorials (on the side bar). Each tutorial targets one aspect of the tool. The reference documentation describes options in detail.

We typically write these tutorials as we implement new features, so that they are as up to date as the latest code. While MPS is different, having a space to post tutorials about the latest MPS features would make it easier for everybody to find what they are looking for. Once the feature is not so new anymore, the tutorial still can be found by keyword and provide useful info that beginners do not know.

I agree - step-by-step tutorials are great for beginners, but having these design discussions and comparing design alternatives is invaluable for people trying to learn how to leverage MPS on real-world projects.  As Federico said, I see this happening through a series of blog posts.

I think we should begin the blog posts as soon as the site is up and running, which I hope to have done today or tomorrow.  That way, people can read through the posts in order and really see how the project evolved, why certain decisions were made, what alternatives were considered (and when they might be appropriate), etc.  I'm happy to take the lead on this by creating the first few blog posts, and then others are welcome/encouraged to join in as we get going.

A few other housekeeping matters:

1) I saw no objections to using MPS 3.0 EAP for the project, so I'm going to just go with that.

2) Thank you to everyone offering to donate their languages. One difficulty myself and others have had with finding languages to use is that so many that are out there are out of date or have been abandoned.  Combine this with the unfortunate fact that MPS is a horrible name to Google on, and it is generally difficult to track down relevant info on MPS, and when you do find it, more often than not it's out of date.  Of course, these are the types of problems we are trying to fix with this community site.

I have some ideas that might alleviate some of these problems regarding languages, but it will require more commitment from the community.  We can discuss all this later and decide what approach to take.

EDIT - took out some options I listed for the site.  I found how to do our blog with Github pages and Jekyll.  This will be quick for getting started.  Then, I think we should start building our languages so that the site is generated using our DSLs, and of course with that in place we can more easily change backend systems later on (and of course, blog about it in the process).

Ideally I would produce series of blog post with the resulting code committed with a tag.

I could start producing posts on implementing JSON with custom persistency.

Of course I am expecting some suggestions to improve it and learn more about MPS!

For the blog I would start with the easiest thing possible. We could later improve it :)

These languages of yours seem quite interesting!

Of course I am expecting some suggestions to improve it and learn more about MPS!

So do we all, I think. Lets see how code reviewing works with MPS...

What if we would simply start a blog on wordpress (free, easy and it is quite adaptable)?
Then we could start with a list of topics to write about, a sort of "editorial plan" for the next weeks?
What do you think about that?

Yes, let's get started eventually. I think I would like to start contributing my tiny features samples and blog some documentation about them.
I see also one our two reports where MPS was part of industrial projects and how it did help to solve problems.

I would start it on "jetbrains-mps-community.wordpress.com".
Could someone suggest a better name?
"metraprogramminsystem-community.wordpress.com"?
I am afraid mps could be confused with many other names and I feel something like "community" or "users" should be in the name.

Suggestions are REALLY welcome :)

I apologize for being out of the loop the last few days, and being tardy with getting the blog up and running.  The former was due to a family emergency, the latter due to thinking through how I want to approach all of this.

As I mentioned before, I have been wanting to do this for a long time, and as I start working on things I realize I might want to stick with some of my original plans, which are not entirely MPS-centric.  I'm certainly not against collaborating - far from it - but my goals are a little bigger. I think getting the MPS community jump-started is too important, and I don't want to hold back this effort if pursuing my other goals interferes.

In a nutshell, I've been planning a blog/tutorial site which I've tentatively titled "DSLegance" (for dsl and elegance).  MPS is certainly a major focus, but I also want to over time cover other areas of DSL development, particularly Xtext and internal DSLs using either Scala or Clojure.  I have no problem focusing on the MPS portion first and doing the other stuff (on my own, if needed) later.  Maybe what I need to do is create my blog separately, but I can cross-contribute content on both blogs, or something like that.

I've been putting together notes on an extended tutorial series using a blog site as the example, and evolving that from a simple static blog to a dynamic, data-driven blog.  Along the way are numerous opportunities to demonstrate all the MPS goodness in a structured way - the more I work through the plan, the more excited I get.

I use Evernote and have put all this in a separate notebook that I am willing to share as soon as I finish the first draft of my plan.  I can give you the link to the notebook and you can look over what I am planning, and based on that we can decide how to best collaborate.