How can we keep the development of HLS sustainable?

[michaelpj]

This is a summary of some discussion that happened on IRC, prompted by @pepeiborra saying:

We are seeing that most HLS plugin contributors find it difficult to provide continued support for their plugins shortly after donating them to the HLS Team.
Maintaining HLS/ghcide is hard work, adding all the contributed plugins to that is a bit much. We will have to figure out a better model going forward.
I don't know if it's more summer projects/bounties, a semi-official social contract like Stackage does, or a technological solution based on dynamic code loading, but we should probably start figuring it out.

Apologies if I misrepresented anyone, please correct me. Also plenty more to discuss.

Compatibility work

@michaelpj claimed that a lot of the painful maintenance version is due to compatibility work, e.g. upgrading to support new GHC versions, or versions of downstream libraries. The contributors of plugin code aren't leaping to contribute to this work, since it's unpleasant and difficult and they're not forced to.

It would be good to spread this out. A few ideas that came up:

• Ask the HF to pay for someone who knows GHC well (e.g. from WT) to do occasional compat work for us.
• Encourage plugin authors to feel enough ownership of their plugin that they do the compat work. See below for how we might accomplish this.
• More summer projects or bounties.

Plugin ownership

As @pepeiborra observed, we often see people contribute (good!) plugin implementations, and then largely lose interest in HLS and not contribute after that. Ultimately this means that the maintenance burden falls on the "core contributors" (here just: anyone who keeps contributing!).

It would be helpful if we could encourage people to stay engaged, perhaps by making them feel that they had more "ownership" over their plugins. Some ideas for how to do this:

• Split plugins out into their own repositories. See below for more discussion of this.
• CODEOWNERS in the HLS repository.
• Leave hackage maintainership with the original authors, rather than @jneira doing everything.

Splitting out plugins

Putting plugins in their own repositories would potentially help, since the repository owner would feel more obvious "ownership" of the codebase, even if some other HLS maintainers also had write permissions.

This would come with all the usual tradeoffs of monorepo vs polyrepo development: doing large cross-cutting changes (e.g. aeson-2 support) would be more difficult and require multiple PRs to multiple repos; Hackage releases would be more spread out; code quality might be more variable etc.

Runtime code loading

@jneira argued that we really should move towards runtime code loading of plugins, truly removing the build-time dependency of HLS on its plugins. This is philosophically neater and would avoid issues with circular dependencies between repositories (haskell-language-server -> my-plugin -> hls-plugin-api).

@michaelpj argued that

• Runtime code loading brings lots of problems of its own
• Runtime code loading in Haskell might well require everything to be built with consistent dependencies anyway (note: is this true?)
• Having an "open" set of plugins known only at runtime will make life very difficult downstream, e.g. in the vscode extension (what options to show?)
• Splitting plugins into their own repositories would provide much of the benefit: HLS could depend on plugins conditionally as it does now. The integration work would consist of checking whether a plugin works, turning it off if not, and waiting for the maintainer to fix it and turn it back on.

[alanz]

My original concept was always to have a very lightweight main file which is autogenerated from some local configuration.
This local configuration is basically the equivalent of a cabal file, in that it lists the plugins that are to be included in the generated main file.
Then HLS can ship with some core set of curated plugins, and it is up to individual users, or distributions that will undoubtedly arise, to curate other sets of plugins.

So in the current source, exe/Plugins.hs gets autogenerated from a script, which also generates an appropriate build setup (cabal file, nix config, stack file, cabal project, whatever).

This does introduce the problem of not having a "standard" one to download.

[pepeiborra]

Users can already build a custom HLS using Cabal flags. Currently all buildable plugins are enabled except for the rename plugin, which is deemed experimental and is disabled by default.

So modulo the code generation of exe/Plugins.hs, I think that your vision is already realised!

[alanz]

I agree the architecture is there, but the part about autogenerating a package to be built by a user is not.

A way to solve the maintenance problem would be to create something like ghcup that allows a user to select the plugins they want to have added to the bare-bones hls, and it generates and builds a custom package for them, as the hls to be used on that machine.

This is the layer that is currently missing, and because of it the burden on HLS maintainers is likely to grow without bound.

[pepeiborra]

To do this in ghcup we'd need to keep a mapping (HLS version, GHC version, Plugin) -> Boolean. But this mapping already exists in Hackage.

So cabal-install coud be this tooling, if we change the plugin flags to Manual:false and ensure that plugins have tight bounds. The only thing missing is a friendly UI.

What do you think?

[alanz]

I wrote a long answer, and then deleted it as irrelevant.

So the idea is to have the plugin enabled automatically on a given (HLS version, GHC version, Plugin) triple when the HLS executable is built.

Which means that building for GHC 9.2.1 will initially ship a minimal set of plugins, but will not be held back waiting for the full set. And this does not affect building the same source for earlier versions of GHC, which will receive the full set of plugins.

That sounds workable.

[michaelpj]

Which means that building for GHC 9.2.1 will initially ship a minimal set of plugins, but will not be held back waiting for the full set. And this does not affect building the same source for earlier versions of GHC, which will receive the full set of plugins.

But that's essentially the state that we're in today, no? The toggling of which plugins are enabled is a bit too manual, but mostly because we weren't brave enough to set the cabal flags to automatic ;)

This does introduce the problem of not having a "standard" one to download.

This is pretty bad IMO. Our users love having static binaries to download; my suspicion is that they would chose a non-configurable but downloadable version every time. And those who care can already compile their own.

Plus, I really do think that an open set of plugins is bad for the UX. Again: how would the extension know what options to offer? Do we write some kind of complicated option-negotiation API with the server? Things are much simpler if we just know the set of possible plugins.

[Anton-Latukha]

HLS <-> Plugins <-> all Plugin dependencies tree - have two-way coupling.

Two-way coupling is a notorious characteristic in architectural design, because it is most frequent example of strict rigidity in the systems which is additionally known to propagate the strain not only through itself, but also around itself. I showed that HLS build depends on the builds of pluging dependencies. HLS build strictly depends on .project, which strictly depends on build of every mentioned plugin, build of every plugin strictly depends on its tree of dependencies (including HLS as a dependency & we see here the two-way coupling literally, right here). Any of the tree of plugin dependencies falls - the HLS can not pass the build. & so the change in brittany - suddenly, through its plugin - created problems updating the HLS to work with 9.0.1 - here we observe the infrastructure strain/requirements goes upward (from plugin to HLS) instead of downstream.

Change in brittany (only new version supporting GHC 9.0.1 & requiring aeson 2.0 at the same time) suddenly required (from jneira) to readjust the whole HLS infrastructure to it (a bunch of .project files & the CI configuration) & debug the situation & essentially do a big trip around to other authors & maintainers, of plugins also, to make requests to them, negotiate & do a bunch of work. And it was mainly not to support brittany, the main agenda of that was in the scope for HLS to be able to release support for GHC 9.0.1, aeson 2.0 & britanny was implying additional complexity & requirements in it just because of the two-way coupling.

We can observe the root of brittany two-way coupling situation in this comment: #2473 (comment).

Where the superficial project dependency, brittany (dependency of brittany-plugin, clashed with the core dependency of HLS, lsp-types.

To be clear - it is not because of particular package, it is just real world example of superficial to the agenda introduced complexity, because of the .project coupling with plugins.

It complicates the process, instead of HLS having the ability to simply think & address GHC 9.0.1 API <-> HLS support of it & HLS own direct dependencies & then making the release. The release of HLS in itself would motivate the community, plugin authors/maintainers to start working on their parts of the infrastructure.

ghcup already has recommended tag, so HLS can release faster & early without plugin support & community activates for their favorite plugins to support new HLS release & plugins update to new requirements & then when enough plugins are adjusted - HLS version gets the recommended status, or some additional tag, for example, "plugins ready", to indicate where the quota of plugins updated to new HLS release.

There is also the question. Currently support of API changes in plugins lays on the who does the HLS code changes. Would plugin authors be able to address plugin API HLS updates - is to their experience.

[pepeiborra]

Can you summarise this into a crisp set of learnings and/or recommendations?

[michaelpj]

And it was mainly not to support brittany, the main agenda of that was in the scope for HLS to be able to release support for GHC 9.0.1

I'm not sure that's strictly true. We could have had 9.0.1 support by just dropping the brittany plugin for that configuration. @jneira chose to just go ahead and fix it.

I still think the workflow would be fine if plugins were in different repos and we just didn't include them if they didn't build, leaving it to their maintainers to fix that.

[Anton-Latukha]

Yes. Theoretical elaboration is just to show in theoretical terms, using example at hand (brittany), that currently whole HLS repo is coupled, currently through .project & CI requirements (that all plugins must pass build & testsuites). So HLS repo is currently 1 structural unit, which would've been bearable if not plugins & their dependencies graph, which is wide & large & involves with HLS core machinery processes.

Of course, the enabling/disabling plugin is several lines away. But that management is done not through community, but almost exclusively by only a few people who have enough project authority to take project development as a priority over project partition consistency. Change of plugin switch is required to be included into 1 dev cycle of the main maintainer. Neither I nor most of the other people are bold enough, during dev, to also "just disable" & ship disabling some plugin that broke during update HLS work, that task of dropping some plugins from the current process, is mainly gets offloaded to the decision of main people in the project.

There is a big number of plugins & they constantly get involved with their build requirements into the main project development processes.

The idea is that main maintainer ideally should not babysit the state switches of plugins at all. Or at least the state of plugins should be only consideration during development, as there is a theoretically infinite number of plugins & they can have mutually excluding requirements. CI should test for plugins, but their return state should not be a strict block. Because the more plugins would be in repo - the less likely all of them would be satisfied at the same time. Either the devs should be explicitly asked/allowed to disable plugins in PRs, with code comments why it was disabled - that would offload some of the plugin management onto the developers who during development are in the context to understand what changes broke some plugin. But that is an ad-hock solution, the architectural solution is preferable. HLS core development should be able to work without worrying about plugins, monitoring of plugin states should be the option.

[jneira]

Well to be precise, brittany already was excluded for ghc-9.0.1. The 9.0.1 support release was mixed with aeson-2.0 and we could keep it out of hls.
But it only would delay the adoption of aeson-2.0, already started in lsp so we should have done it anyway

[pepeiborra]

I'm going to address the points summarised by @michaelpj in individual posts, starting with this one.

Compatibility work (GHC)

Indeed, this is the main source of maintenance overhead, and the initial motivation for this discussion. The 9.2 GHC upgrade has been a huge effort and is far from finished, with only 9 out of 19? plugins migrated. I stirred a discussion with the hope of getting the attention of plugin maintainers and coming up with a plan to make future upgrades easier!

Now, plugin contributors cannot really upgrade their plugins until the core HLS packages, namely ghcide, hls-plugin-api, hls-graph,hls-test-utils, etc. have been upgraded, so it’s in our best interest to perform those upgrades as soon as possible, including the upgrade of dependencies.

Proposal: invest our OpenCollective funds in summer projects or GHC consultancy to help with this work, as done last summer by @fendor.

[michaelpj]

+1 to some version of "solve this problem with money"

[michaelpj]

(I also love monorepos!)

[Anton-Latukha]

Now, plugin contributors cannot really upgrade their plugins until the core HLS packages, namely ghcide, hls-plugin-api, hls-graph,hls-test-utils, etc. have been upgraded, so it’s in our best interest to perform those upgrades as soon as possible, including the upgrade of dependencies.

Would restate from: #2588 (reply in thread)

HLS core development should be able to work without worrying about plugins, monitoring of plugin states should be the option.

Before that: "HLS release would motivate plugin authors & community to update plugins to new release".

[pepeiborra]

Plugin ownership and individual repos

What benefits does HLS get from the current monorepo structure?

1. API changes never break plugins. However, we still manage to break Hackage builds due to incorrect version bounds or version bumps
2. Binary releases require little coordination: no need to chase plugin owners for a Hackage release, or source repo overrides for plugins, or Hackage snapshot bumps
3. Core changes receive extensive testing - since CI runs all the plugins test suites.

The hope is that individual repos would lead to an increased sense of ownership, presumably because issues, upgrade requests, and volunteer patches would be reported directly to the plugin repo, which would nominally belong to the plugin author. But that’s assuming that users are able to identify the plugin and find the plugin repository easily.

No proposal for now, since for me, the benefits of the monorepo are just too good, and the assumption that users will be able to find the plugin repo is dubious.

Is there any solution that preserves the monorepo but increases the plugin ownership? Perhaps an informal contract like that of Stackage?

[michaelpj]

Suggestions:

• CODEOWNERS
• Putting plugin authors as the maintainers in cabal files
• Some kind of informal contract could work, maybe just something like "if your plugin is accepted, you agree to maintain it, otherwise we will aggressively drop it"?

[jneira]

Suggestions:

* CODEOWNERS

👍

* Putting plugin authors as the maintainers in cabal files

👍
(and puting them as hackage uploaders)

* Some kind of informal contract could work, maybe just something like "if your plugin is accepted, you agree to maintain it, otherwise we will aggressively drop it"?

Not sure if that fits the open source culture (well at least not mine one). I think it is prefectly valid if someone wants and have time to contribute at some point.
Drop the plugin is a thing always can happen. We could ping plugin authors and decide in each case.

[pepeiborra]

Runtime code loading

If we split the plugins out, binary releases become much more of a headache. Runtime plugin loading offers a solution by allowing binary releases to essentially ignore plugins, and hand over their management to end users or other tooling like ghcup. But this benefit comes with a set of problems/costs, well identified by @michaelpj.

No proposal since it doesn’t look too promising, although that could change if someone develops a working POC.

[jneira]

Agree with pepe's thoughts. Only note that we have to deal with problems affecting the haskell development in general. Aggravated by the fact we are deep downstream in the dependency graph and our dependency set covers "half of useful hackage". So generic solutions already discussed in other community forums will benefit us more too.

And we are totally tied to the ghc api and we have to deal with the continuous stream of breaking changes coming from ghc development.
But we have to take in account several ghc maintainers are helping us to do it (thanks all!), dont know how could we tackle it otherwise.
Anyways having a good compat layer with ghc would help us a lot to both core and plugin maintainers.

Not sure if move the code to other repos could make the situation qualitatively different.

[alanz]

Not sure if move the code to other repos could make the situation qualitatively different.

So maybe we should agree that the monorepo is a good idea, and consider how we can decouple things so we have a core that has to work, encompassing some minimal set of plugins, and then HLS proper which has the full or carefully curated plugin set.

So some sort of arrangement similar to what we initially had where ghcide was a standalone project, that we used in HLS.

Given the ability to configure projects in cabal (and stack), we should be able to come up with a usable partition, and then arrange for the CI to run differently for the core, and give the appropriate signal for it.

[jneira]

Well i think the actual internal dependency set already is the proposed by @Anton-Latukha and @alanz in his last comment:
{ hls-graph, hie-compat, hiedb, hls-plugin-api, ...} -> { ghcide } -> { 0..n plugins} -> { hls }

• hls is a really tiny layer for ghcide and plugins, if you disable all plugins hls is almost equivalent to the old ghcide executable
  • so we could offer a hls version as soon as ghcide is ready for a specific ghc version
• as for plugins i think broadly speaken there are two kinds:
  • plugins which only (or in practice) depend on ghcide and ghcide dependencies (notably ghc). For example, hls-module-name-plugin, hls-*-imports-plugin, etc. Those ones could have lived in ghcide itself and update them after having a working ghcide is relatively easy. They dont have extraneous dependecies which can cause dependency hell.
  • plugins which depends on ghcide/ghc and other external library with its own dependency set, standalone executable and, more importantly, agenda: all formatters, hlint, retrie, etc. These ones gives us more headaches and in fact we are thinking on them when we complain about having to keep plugins up to date.
    • These ones can be happily excluded from hls and they had been excluded for ghc-9.0.1 for long time.
    • But as far we want to give our users the most complete and satisfactory ide experience possible we should try to include them again asap
    • Otoh authors of the underlying lib should be eager too to get the library available via hls cause a good part of their users comes from hls and not only the executable nowadays (and that will be so more and more)
    • So an option could be move those plugins to the repo of the lib, like the lib executable. Afair we talked about that possibility in the past.
    • The dependency set of lsp could be really different to the existing lib one so it should have a separate build plan.
    • I wonder what would think @ndmitchell about moving the hlint plugin to hlint, or @tfausak about move the brittany plugin to the brittany repo

[michaelpj]

Otherwise, I agree that we basically have this setup today, if we were just more aggressive about disabling things and releasing without them.

[jneira]

ugh, i wanted to mention @ndmitchell 🤦

[Anton-Latukha]

Yes.

I pretty much stated the main structural point & shot my mouth up, because I do not know enough for the project, why it is considered impossible to separate the plugins as they are & their maintenance from the HLS.

As I proposed, plugins should be kept inside haskell. As the creator of the package - is not actually responsible for the plugin creation & maintenance - those can be quite different people. Maybe the author does not have time to maintain plugin. If the package is private project - the HLS plugin is a community effort to use that project in HLS. Keeping the plugins in haskell group would allow having main core of plugins & their maintenance/development in one group, haskell group would be able to access them when need arise to fix them, or maintain. Authors still can have proper access to plugins.

Also remember I talked about upstream monitoring/notification support for collaborative plugins in haskell group. The inclusion of plugins into haskell group - would mean solving the bus factor for each such plugin, also allowing to signify importance & a minor agreement with authors, to users indicate that plugin is well supported & inside HLS it would be easy to track & monitor them & notify them before release if upstream API changes occur. Otherwise, when there would be 100 plugins - it would be hard to remember wich plugin is considered in the good state (& should be monitored) & which is in the bad state & should go out of the monitoring & transferred to their license holder/most recently active author.

[jneira]

I get the point of moving (or at least create the new ones) plugins to the haskell github org. But maybe they will not get the required attention precisely for being in the organization without a clearly known and assumed maintainership.
I think it is easy to attract and engage people wanting the plugin working and up to date around hls or the library powering the plugin.

[Anton-Latukha]

I personally take group projects more seriously than personal ones.

I always do want to ship good work & keep the project working, if I took responsibility/activity for the project in the group. If I keep the project in a problematic state or in work-in-progress draft state in my private account - that is a sort of my private property & my business, but the public groups mean that a lot of people observe & even have a passive potential circle of participation, and the active one - if I would put/something would break the project & I would not address it in the public group - I know I have not done it & so feel uneasy.

Long story short. I think it is prestigious to maintain the project in the haskell group & from my point of view that motivates people maintain more, then to maintain the project in somebody's private account which can be perceived as "doing work instead of them", "doing work for them" & "for their benefit", or sending plugin to the author frequently would mean anyway that only author would maintain it.

Additionally - would reverse the previous implication arrow: it would be hard to have a recommended or extra catalog of plugins - when HLS loses the track of which plugins even exist & where, since they all would be in 3rd party repos. The awesomeHLS repo would be required to exist & be curated continuously.

[eddiemundo]

One thing is notifying the relevant authors of some plugin, or having some kind of issue/table/system saying X plugin needs work. I just kind of assumed that the plugin I wrote worked in 9.2, and then I saw a pull request of @mrgutkun fixing it and was like oh oops you guys were talking about me.

[jneira]

Well, i did not follow closely the pr about ghc-9.2.1 support although i did some auxiliary commits trying to help people working hard on it.
Only after merging it i filled the issue about ghc-9.2.1 with all the checks, checking the build and ci config files (the truth always is in the code 😉). And finding some mising things in the way, so a work that worths.

Maybe having those checks beforehand could help, but, well, keep that up to date is a work in its own.

[eddiemundo]

I definitely missed that issue... I think what I'm saying is subsumed by the "CODE OWNERS" suggestion. I'd be OK with being pinged for maintenance.

[michaelpj]

Great idea, a new one just occurred to me: can we make a @plugin-maintainers github team? Then we could mention it in e.g. the GHC version support tickets, or if there's something that needs input from plugin maintainers in general.

[michaelpj]

(answering my own question, seems like we can create such a team easily enough if we want to)

[michaelpj]

Okay, concrete suggestion: I propose to:

• Add a CODEOWNERS file with plugin authors associated with the given plugins, and configure it so that they are requested but not required to review PRs touching the relevant bits.
• Create a hls-plugin-maintainers Github team with the same members.