Dr. Jan Köhnlein is one of the founders of TypeFox, the moving force behind Xtext, Xtend, and FXDiagram. This is a very extensive follow-up interview to this one. Thanks again to Jan!
Dear Jan, thank you very much for your time. I really like the ecosystem around Xtext, Xtend, Xcore, Xbase, and EMF. I think it is one of the most-disrupting decisions (in a positive way) that, starting from 2.9.0, you address more than just the Eclipse platform! But with this new approach and your all new company TypeFox (as well as the spin-off from itemis), some questions arise…
In your interview with Frederico Tomasseti you mentioned, that the response from the community regarding your IntelliJ-Plugin is not as expected. What do you think are the reasons for that?
I can only guess. I think that it is not so common to install extra plug-ins in IDEA as it is in Eclipse. Users seem to stick to the default IDE config and rather integrate any “additional tools” in the external build. And of course Jetbrains’ Kotlin language comes with better IntelliJ integration than Xtend does. To be fair I must admit that the quality of Xtext/Xtend in IDEA is a bit behind the Eclipse version.
I personally tried to use Xtext for IntelliJ, but had some issues right from the beginning…
I added an issue as well as another one to bugzilla in february, which is unanswered (status NEW) until now. The issue tracker seems to be full of “NEW” issues. In several articles Sven Efftinge said, that the highest priority is currently to stabilize the project. So why are so many issues unanswered and will this change soon? Might this be one of the reasons for the missing response to the new IntelliJ feature?
Let me first excuse that we haven’t been as responsive recently as you might expect. Founding a new company is quite an effort, and because TypeFox had a very successful start, we already have a lot of customers to serve. But we are gradually getting back to more open source activity.
What Sven meant by “stabilizing the project”: We’ve heard quite often that Xtext was considered a bit too heavy weight. So we’ve spent the last weeks splitting up the Xtext github repository into separate sub-modules for the various usage scenarios: the core, Eclipse, IDEA, Web, etc. As such, e.g. a user who only wants the Web integration does not have to bother with the details of Eclipse or IDEA. You can even just clone a very lean version of Xtext if you don’t need any IDE support. That was also quite an effort and kept us rather busy.
The repository split also separates stable modules, like Xtext’s core runtime and the Eclipse integration, from the newer and a bit more experimental ones, like the IntelliJ plug-in. In the future, this will also make it easier for third parties to take over modules they may benefit from more than we do.
As for the IntelliJ part, we really spent a huge load of work to provide it in version 2.9. IDEA turned out to be a moving target and we had to spend another pile of work to keep our codebase compatible. It took until recently until a first customer mentioned interest in porting their DSLs to IDEA, and thereby maybe fund further development and bug fixing. This is why it has been a bit silent from our side regarding IntelliJ.
A possible alternative to the current IntelliJ support may come via the Visual Studio Code Language Server Protocol.
How is the reception of the target web for Xtext?
We have customers using the Web integration. With Ace, Orion and CodeMirror we build on pretty stable APIs. But as opposed to the IDEA stuff, the Web integration is a bare editor. Defining some kind of workspace concept, e.g. to allow references across resources, is up to the users, and that is much more involving than just installing a plug-in in the IDE. This is why we also look at Eclipse Che.
I have some questions regarding IntelliJ support:
- Did you create the IntelliJ Xtend plugin completely with (the new) Xtext 2.9.0, in order to have a “big example/proof-of-concept” like you have in Eclipse? Or are there still some limitations, that had to be circumvented?
- Are there plans to unify the build management approaches (maven, gradle, …)? The current setup (seems to) prohibit(s) sharing code between an Intellij and an Eclipse project (or am I wrong here?).
As we started from the very same grammar as in Eclipse, and as IDEA does not really allow to work on Eclipse plug-in projects, we had to generate the IDEA plug-ins from within Eclipse. Same holds for Xtext. So we didn’t really bootstrap Xtext and Xtend in IDEA, but used Eclipse. Nevertheless you can import and maintain the resulting projects in IDEA. And of course you can use only IDEA to build your own DSLs.
And yes, the unification of the build process is already in progress. While splitting up the repository we also switched to a plain Gradle build for the Xtext/Xtend modules. We use Maven/Tycho only to create the Eclipse artifacts (plug-ins, features, update sites) from the Gradle compiled binaries, as Gradle cannot do this yet. This should make it far easier to consume Xtext for non-Eclipse users, and the Eclipse build is no longer an entirely separate thing. The same mechanism can be applied to our users’ DSLs.
Are there plans to support Xcore via maven in Eclipse and in IntelliJ?
I think this would lower the entry hurdle (using EMF) enormously. I really, really like to use Xcore (which is part of the EMF code base) for defining a meta model for my DSLs. Auto-generation of a meta model from a grammar is not always suitable (especially, if you use the meta model somewhere else and want to use EOperations). Since you are promoting DSLs, adding Xcore to the core elements (what a pun) of Xtext seems to be a natural step, doesn’t it?
I like Xcore a lot for the same reasons. But it puts a bunch of additional dependencies on the runtime classpath of your project, which you might not want. BTW, you can already use Xcore with Xtext, which usually boils down to loading it as referencedResource in the MWE2 workflow of your language and add a few dependencies. Maybe I should write another blog post on this…
Xcore is part of the EMF project. So if you want an Xcore editor in IntelliJ, you could to contact the EMF committers.
I would really appreciate such a blog entry. I think it would help a lot of people.
My experience with Bugzilla was so-so. Do you plan to move to GitHub for issue tracking, like you did for your code base? I had a look at the GitHub issue trackers of Xtext-core and friends and they are quite empty (especially in comparison with the corresponding Bugzilla tracker).
All the cool kids are on Github, aren’t they? Joke aside, having our entire code base on Github, using Github issues is the a logical step forward. We already support them, and we file all of our new tickets there. But we’re still figuring out which tags etc. to setup to get on par with Bugzilla. We haven’t decided yet whether we will deprecate Bugzilla some day and how to deal with the existing tickets then. So currently, you can just use the one you like better. If in doubt, be cool and use Github 🙂
Who does not want to be cool? This is why everybody is in the kitchen on parties right?
Is itemis still supporting Xtext after the split-off of TypeFox?
Yes, itemis keeps committing actively to the Xtext and Xtend. By the way, there are also active committers from other institutions, like Lorenzo Bettini from the University of Torino. We really enjoy this diversity. It’s very productive and we’re all on good terms.
Are you to busy with building up the new company, handling all the customers, and fixing bugs or isn’t the site of TypeFox meant (together with Xtext.org) as the main hub for information?
The blog and the site itself do not have any posts newer than march (except one about the language server protocol on june the 30st). I would have expected some success stories of customers there and some examples where DSLs can be useful.
Unfortunately, not every success story is a public one 😉 You are right, we’ve been quite busy recently and neglected writing blogposts, but that is going to change. The websites of TypeFox and Xtext are still the main hubs.
The Visual Studio Code Language Server Protocol that Miro blogged about is in fact a game changer and we consider it extremely important for the future of Xtext and IDEs in general. A language server represents the language specific backend for a generic IDE. Think of it as a compiler with additional support for content assist, hovers, etc. Language servers communicate with IDEs clients, like originally Visual Studio Code, via the language server protocol. This allows to separately implement a generic IDE GUI and backends for various languages. It has the potential to eventually enable all languages in all IDEs! We have implemented the language server protocol in Java and based on that we are building the backends for Xtext DSLs, Xtend etc. At the same time, other companies are already creating clients for IDEs like Eclipse, Eclipse Ché etc. IntelliJ would read good on that list, wouldn’t it?
As quite a few members of the original Eclipse JDT team are now involved in the design of VS Code and the language server protocol at Microsoft, we are pretty confident it will get some momentum.
Did you already contact the guys from Jetbrains?
Perhaps working together (join forces) could be very beneficial for both sides (I know they have 850 employees, but Google bought Android from a small company way back then 😉 ). Especially, as they have their own open-source project for DSLs the MPS and as I heard, they seem to be eager to combine their projectional editors with normal parsing (using antlr).
Of course, we have been in contact with JetBrains developers ever since we started working on the IntelliJ support for Xtext/Xtend. They are very cooperative and responsive, and it feels like they would like it if IntelliJ became a community platform like Eclipse already is. I think, MPS targets a different audience, so I don’t see too much competition with Xtext. Kotlin somehow competes with Xtend. Long story short: We collaborate but never got as close as “joining forces”.
I saw that you support standalone Xtext (without IDE), too. Do you expect that there will be Xtext-only IDEs completely independent from established IDEs like IntelliJ or Eclipse? Do you think that would be beneficial (since Xtext benefits from the respective ecosystem…)?
Yes, there will be. We are running experiments with Eclipse Che, VS Code and other new IDE platforms that allow to plug in languages (and plug out unwanted ones). I expect to see many of these special purpose IDEs in the future, especially in the cloud. The language server protocol could help to make that affordable in cost for both, development and maintenance.
A blog post from June 2015 of you shows, that the project FXDiagram will be available not only for Eclipse, but standalone, and an IntelliJ version, too. Will TypeFox support this fully? And what is the timeline here?
The core and most other parts of FXDiagram are independent of Eclipse, even indepentent of an IDE. In fact FXDiagram it runs a bit smoother in Swing based applications as IDEA than in SWT (Eclipse) because of the better JavaFX bridge. Unfortunately, Swing does not support multi-touch gestures 🙁 . The code for a class diagram editor in IDEA is in the repo. It shows how to connect to IDEAs internal PSI model. Have you tried it? There is just a little bit more tooling support in the Eclipse version, but that could be easily ported if someone shows interest. Of course, contributions are welcome 🙂
I did not try it, yet. But I am excited to check it out.
Do you follow a hard release schedule like some agile projects (e.g. GitLab)? Or what is your release modus?
We are riding the Eclipse train, which means a new release every June. We tend to have another release before EclipseCon Europe usually in October. As we can’t work full-time on open source, we don’t have any more fixed releases, but schedule additional ones when we feel we have created enough added value. When we encounter blocker bugs, we fix them in maintenance releases.
What is your testing strategy?
This seems to be a challenging task for your projects.
Of course we have a continuous integration as every modern software project should. We use checked commits on Github, i.e. a PR has to pass a large set of unit tests before it can be merged. There are several tens of thousands of unit and integrations tests, some smoke tests and a few SWTBot tests. Before the repository split we found ourselves in a situation where we had too many tests to run on every commit. After the split, the number of tests that have to be rerun after a commit is significantly reduced, especially is you are working on a downstream module. Some tests are only executed every night.
Before each release, we perform at least a week of manual testing.
In other interviews Sven Efftinge and you said, that the business model of TypeFox is more or less about consulting/training. Do you have further plans?
Xtend is a full-fledged language with unique features and does not have to hide behind other modern languages. Further on, your eco-system is very bootstrappy, systemic, and holistic in that it ties together language and IDE development as well as (major selling point) becoming domain-specific. I know development tools are hard to sell, but don’t you have some kind of product to sell in mind?
Currently we are working mostly on products from our customers, and after several years of consulting with lots of travelling we like the fact that we can work mostly from our office or from home. Some of these customer products will go public and have a good potential of becoming successful, so we are already in the product business somehow. We do have a couple of own ideas and a list of potential partners as well.
Do you sell extended versions of your tools (which are not open source), do you adapt your tools for your clients (so that these parts are not open source), or is your philosophy that every change to Xtext and co. is integrated into the open source products eventually?
Whenever we implement a new additional feature for a customer, we try to feed it back into the open source codebase. Most customer welcome this way of giving back to open source. In very few cases we’re bound by NDAs that forbid that. We don’t have and don’t plan any commercial version of Xtext or Xtend.
Do you have a community process for evolution of Xtend, Xtext and so on?
Languages like Java, Kotlin, and Swift have a community process for language proposals. Do you have something similar for your projects, or how should someone approach you, if he or she has a good idea and perhaps even an implementation?
That would of course be nice. but requires more work to keep the community going than we as a small company can currently spent. For enhancements, it’s best to file an issue on Bugzilla or Github or discuss it in the respective user group. And if you have some code to spend, even better!
Sounds great. I have some exciting proposals I would love to share (and implement).