New plugin API: Inter plugin communications

[A-UNDERSCORE-D]

Looking for what all the plugin devs want when it comes to an inter-plugin communication API, to roll into the new plugin system Im slowly building at https://github.com/A-UNDERSCORE-D/EDMarketConnector/tree/feature/new-plugin-system

Currently Im planning to already have a much more open event system, where plugins can fire arbitrary events and hook on arbitrary events, rather than only allowing ones that are programmatically built into EDMC, but I digress.
EDIT for clarification: The event system mentioned above is the "same" one that will carry new journal updates and so on, its unrelated to the IPC discussion below, where the above events are broadcast messages, and IPC is 1 -> 1 communication.

For IPC itself, I'm of two minds.

1. Structured RPC system, where plugin A registers a function that other plugins can call. Similar to how factorio does its mod RPC.
2. Simple message passing, where A can say that its happy to receive messages, but the format of those messages isnt defined, and will be passed between plugins as strings.

EDIT 2: Seems people are leaning towards simpler message passing

[lunarplasma]

I would suggest that the second approach (simple message passing) is more open-ended, and probably simpler to both implement and utilise. I'm interpreting it as a system where each plugin can "broadcast" its data and any other plugins can choose to listen to broadcasts - if it gets a message which it wants, then it will do something with it.

However, I'd say that passing around JSON would be better than simple strings. Even better: dicts. The plugin source name could be one of the keys.

[A-UNDERSCORE-D]

The broadcast idea I was thinking would be more covered by events rather than IPC, a plugin can chose to fire an event with a name (likely prefixed by the plugin name to ensure there are no conflicts) and other plugins could hook a callback onto a given event

[A-UNDERSCORE-D]

I realize I didnt actually answer half of that. Yes dicts would be better
Hell we could define a class too that is used for message passing, contains a user dict + meta around the message. Even better-er we could allow subclassing of that too.

[lunarplasma]

The simplest and most elegant way (and probably most naive) implementation description could be:

1. A plugin_inter_api() function as described by @Athanasius below, which is how a plugin would receive dict messages from any other plugins (or EDMC). They can then parse the message if they're interested.
2. A plugin_broadcast(msg) function which can be called by any plugin. The msg must then be a dict message, which eventually gets to other plugins via the 'plugin_inter_api()`. The message would need to go into a queue, to avoid threading shenanigans.

[anthonylangsworth]

Let me take a step back (and forgive me if this is covered elsewhere) but what use cases are driving this? I am not implying we should not have inter-plug-in comms. More implying that they may help direct us to toward an answer.

Brainstorming types of plug-ins that would need to send or receive messages:

1. Enrichment: A plug-in reacts to existing messages or data and adds more information. For example, if you target a commander in-game, the plug-in may call a web API to determine whether they are a known player killer or "care bear" then broadcast this to other plug-ins.
2. External Data source: A plug-in that pulls data from external sources and makes them available to EDMC-plug-ins. For example, a BGS "tick" broadcast that periodically calls the tick API from https://elitebgs.app/api/ebgs on a background thread then sends a message when a tick occurs.
3. Centralized Configuration: A plug-in that centrally stores and provides a UI for configuration used by multiple plug-ins. For example, there are my BGS-related plug-ins out there that follow BGS-related activity for minor factions in systems. It could broadcast the selected minor faction(s) and system(s) at EDMC start and when changed.
4. External Interface: A plug-in that provides a easy-to-use interface to abstract external systems. The advantage over a library is the configuration UI could appear in the EDMC settings window. For example, a Discord EDMC plug-in. This plug-in could send it a message and the Discord plug-in would post it to one or more configured channels in Discord servers.

There are probably other cases. However, none of them favour either a structured RPC or simple message passing system. Non-functional/quality attributes are also minor. For example, EDMC is not time-critical or memory constrained. Therefore, I would lean toward the simpler implementation: simple message passing.

However, there are other questions beyond structured RPC versus simple message passing. For example:

1. Do we care about the order plug-ins process messages? Do we care about the [Edit: added]order messages are received? Is message handling re-entrant i.e. can I send a message when processing a message? Do we limit the message queue or stack length to prevent loops? This is less around simple message passing versus structured RPC. These are common sources of bugs in message processing systems (and will likely raise many issues here in GitHub).
2. Do we want to manage dependencies between plug-ins? For example, a plug-in that relies on one of the cases above could ask EDMC whether a plug-in it requires is installed then, if not, either fail or gracefully degrade. Modules could be loaded in dependency order. A structured RPC system could help, if and only RPC registrations are in plug-in metadata or otherwise known before loading (e.g. a new "plugin3_prestart").
3. If I want to do inter-plug-in communication, is message passing the best model? Are we looking more for a standardized Observer pattern implementation? I suppose this is what the structured RPC implementation would be.
4. Do we want to send messages to specific plug-ins and not others? Do we need some form of response? This is common where (1) messages are commands or (2) the plug-in wraps an external data source like an API. If so, a structured RPC system would be better because you require registration. I am inclined to say "No" due to the complexity it adds and lack of compelling use cases (at least that I can think of).
5. How do we prevent message ID clashes? This is more a documentation recommendation than governance, like with config.get/config.set.
6. How do we handle breaking changes to event formats? A structured RPC system could handle this through versioning or explicit ID changes. Otherwise, an ID change or embedding a version identifier in the payload is usually sufficient.

[Athanasius]

1. Message handling should not be considered re-entrant. It is verbotten to attempt to send a message in the handler for receiving a message. We could in theory enforce this with a flag in the core code that is the intermediary.
2. Fully handling plugin dependencies isn't something I'd want to attempt. If nothing else it would mean needing to re-arrange the order in which they're loaded. I'd be more inclined to add a "plugin_loading_finished" event/function that is called when the core has loaded all the plugins it found so all plugins know they can 'fire away now'. This does then end up somewhat overlapping with point 4.
3. The simple implementation is that we define a plugin_inter_api() function that will be called with a data dict when any other plugin sends a message. The structure of that dict is then up to the sending plugins, for the most part (we can mandate a sender name and version). If a plugin doesn't care about any other plugins then it simply doesn't have that function defined. If it does then it gets to do the filtering for which senders it's actually interested in (we can provide example code/helpers).
4. One use of the responses is for one plugin to ask if another plugin is actually available. On the other hand, with the simple "all I do is get this function of mine called" the plugin should be designed to fail gracefully if it never receives expected data. Perhaps plugin_loading_finished() could pass in a list of the loaded plugins to help with this.
5. Plugin names are already unique (you can't have more than one NTFS folder in the same location with the same name (he says, wondering about 'streams' and the like, but it would take effort to break this assumption)), so use those as a prefix. We won't care what a particular plugin's 'X' message actually is or means. It's up to the plugin author to document it.
6. Again this is going to be specific to a given plugin. We should keep the inter-plugin API simple. source plugin, version, dict as message, plugin authors decide what's in the message structure.

[A-UNDERSCORE-D]

Pretty much what I said below, though I'd prefer to have plugin_loading_finished be an event over a method that we require naming etc on. Easier for plugin devs to work with it that way. and easier for us to add new stuff etc.

[A-UNDERSCORE-D]

Let me take a step back (and forgive me if this is covered elsewhere) but what use cases are driving this? I am not implying we should not have inter-plug-in comms. More implying that they may help direct us to toward an answer.

My reasoning for this is that we already have plugins making calls out to other plugins in the EDMC core, (you can ask for inara and edsm updates to ping you -- see https://github.com/EDCD/EDMarketConnector/blob/main/PLUGINS.md#plugin-specific-events )

Brainstorming types of plug-ins that would need to send or receive messages:

1. Enrichment: A plug-in reacts to existing messages or data and adds more information. For example, if you target a commander in-game, the plug-in may call a web API to determine whether they are a known player killer or "care bear" then broadcast this to other plug-ins.

2. External Data source: A plug-in that pulls data from external sources and makes them available to EDMC-plug-ins. For example, a BGS "tick" broadcast that periodically calls the tick API from https://elitebgs.app/api/ebgs on a background thread then sends a message when a tick occurs.

3. Centralized Configuration: A plug-in that centrally stores and provides a UI for configuration used by multiple plug-ins. For example, there are my BGS-related plug-ins out there that follow BGS-related activity for minor factions in systems. It could broadcast the selected minor faction(s) and system(s) at EDMC start and when changed.

4. External Interface: A plug-in that provides a easy-to-use interface to abstract external systems.  The advantage over a library is the configuration UI could appear in the EDMC settings window. For example, a Discord EDMC plug-in. This plug-in could send it a message and the Discord plug-in would post it to one or more configured channels in Discord servers.

All of these are options, yeah. But a simple system would cover most if not all of their use cases. My main reason for asking this question is to get a feel what kind of system devs would like, rather than getting too deep into implementation (which I will do below)

There are probably other cases. However, none of them favour either a structured RPC or simple message passing system. Non-functional/quality attributes are also minor. For example, EDMC is not time-critical or memory constrained. Therefore, I would lean toward the simpler implementation: simple message passing.

I missed this in my initial read-through and answering stuff, this was written last. Yeah, this is about my feel. Simpler is better for EDMC, if we do need a more complex system, we can probably implement it later.

However, there are other questions beyond structured RPC versus simple message passing. For example:

1. Do we care about the order plug-ins process messages? 

This may or may not actually matter, it can be completely worked around by only allowing message passing once startup and load is complete everywhere.

Do we care about the messages are received?

Either system will require the recieving plugin to have some sort of callback for recieving messages. As for whether or not we want to send back to the caller "yes they got that" that would be dependant on whether its RPC based or message passing based. Message passing would be best effort, RPC would allow for returns.

Is message handling re-entrant i.e. can I send a message when processing a message?

I'd put this down as "Do as your own risk" style. There are reasons for blocking it and reasons for having it. Which leads to...

Do we limit the message queue or stack length to prevent loops?

No. Loops are bugs that need to be fixed, not worked around. Either by plugin developers if they build looping systems or by us if we introduce loops. The python stack limit will do nicely to deal with that. As for queues, I havent considered that. My plan when I asked for this was for it to be syncronous.

This is less around simple message passing versus structured RPC. These are common sources of bugs in message processing systems (and will likely raise many issues here in GitHub).
They are, hence the discussion around them :D

Im wondering if this is going far too deeply into a complex implementation of what can be a reasonably simple system.

2. Do we want to manage dependencies between plug-ins? For example, a plug-in that relies on one of the cases above could ask EDMC whether a plug-in it requires is installed then, if not, either fail or gracefully degrade. Modules could be loaded in dependency order. A structured RPC system could help, if and only RPC registrations are in plug-in metadata or otherwise known before loading (e.g. a new "plugin3_prestart").

I'd really rather not manage dependencies between plugins, but as there are multiple stages to loading (which I guess I should modify now -- Either a final load_complete event sent to all plugins or similar. Yes that does have order issues, but in theory will at least be deterministic), they should work each other out so long as there arent any hard dependencies at load (which for EDMC I don't see what you'd need that you couldn't error out on after a full load)

Specifically for plugin3_prestart, Part of all of the "New plugin API" is that plugins will be class based. Right now what Im messing with instantiates that class and then calls a load method, but as I mentioned above, this could be modified to call an additional method on all plugins once everyone is load()-ed

3. If I want to do inter-plug-in communication, is message passing the best model? Are we looking more for a standardized Observer pattern implementation? I suppose this is what the structured RPC implementation would be.

I don't know, that's why I'm asking :D I don't want to end up with a message passing system where people may have wanted some more RPC like behavior, and thus end up building their own request-response message pairs.

4. Do we want to send messages to specific plug-ins and not others?

Yes, that's what I'm defining message passing as. If you want to broadcast something, use an event. Any plugin that wants to listen to an event can hook onto it. Yes I'm making a distinction in what is essentially two kinds of event systems, but that helps me with a mental model of how I'm thinking this will all work

Do we need some form of response? This is common where (1) messages are commands or (2) the plug-in wraps an external data source like an API. If so, a structured RPC system would be better because you require registration. I am inclined to say "No" due to the complexity it adds and lack of compelling use cases (at least that I can think of).

I covered this above. I'm not sure. This is why I want to get a feel from devs as to what they'd like.

5. How do we prevent message ID clashes? This is more a documentation recommendation than governance, like with config.get/config.set.

If I'm correct in my assumption that you're referring to plugins A and B both defining ID Z, simple, As is A.Z and Bs B.Z. Unless you see an issue with that that I dont see?

6. How do we handle breaking changes to event formats? A structured RPC system could handle this through versioning or explicit ID changes. Otherwise, an ID change or embedding a version identifier in the payload is usually sufficient.

Is this referring to plugins and their own events? I was considering letting plugins work that out themselves, as they're working together anyway.

I feel I need to clarify that I want both an event system (as in broadcast) and a message / RPC system (as in 1 to 1). The former is already in EDMC, just in a very brittle format, and thus will need to be supported going forward--Though I intend expanding it to support arbitrary events from both sides of the system.

The latter seems like a nice thing to have--Whether or not it'll actually be used or otherwise I'm not sure :D but thats why I'm asking here

[anthonylangsworth]

Some good discussion. Thanks!

Do we care about the messages are received?

I left out the critical word "order" in my original message. I.e. Do we care about the order messages are received? An RPC system would guarantee that but a message-passing system may not. I would say order is important.

Is message handling re-entrant i.e. can I send a message when processing a message?

I'd put this down as "Do as your own risk" style. There are reasons for blocking it and reasons for having it. Which leads to...

Do we limit the message queue or stack length to prevent loops?

No. Loops are bugs that need to be fixed, not worked around. Either by plugin developers, if they build looping systems, or by us if we introduce loops. The python stack limit will do nicely to deal with that. As for queues, I haven't considered that. My plan when I asked for this was for it to be synchronous.

This is less around simple message passing versus structured RPC. These are common sources of bugs in message processing systems (and will likely raise many issues here in GitHub).
They are, hence the discussion around them :D

I'm wondering if this is going far too deeply into a complex implementation of what can be a reasonably simple system.

I am probably overthinking. However, if it is easy or possible to write faulty components (e.g. an EDMC plug-in) in systems we create (e.g. EMDC), people often attribute that to the system, not the component.

For example, a message loop will get a stack overflow error, as you say. However, its cause can be hard to triage and diagnose. If can we write a system to anticipate these types of problems and flag them (e.g. message queues are not getting shorter as plug-ins process them, maximum message queue lengths) it can help not only prevent bugs but give developers a better impression of the system.

5. How do we prevent message ID clashes? This is more a documentation recommendation than governance, like with config.get/config.set.

If I'm correct in my assumption that you're referring to plugins A and B both defining ID Z, simple, As is A.Z and Bs B.Z. Unless you see an issue with that that I dont see?

So we'll namespace the message IDs. That makes sense. This is probably already in your code and I did not see that.

[A-UNDERSCORE-D]

I left out the critical word "order" in my original message. I.e. Do we care about the order messages are received? An RPC system would guarantee that but a message-passing system may not. I would say order is important.

Agreed (also I missed a typo in there, because of course I did) Order will be important yes, for both systems, though less so for message passing. that said, as it'll end up being synchronous it should be such that its ordered first come first serve, with no queuing. So I guess the order is "whatever order plugins were added to the plugin list". Im not sure if we need anything more than that, especially if its message passing based.

I am probably overthinking. However, if it is easy or possible to write faulty components (e.g. an EDMC plug-in) in systems we create (e.g. EMDC), people often attribute that to the system, not the component.

For example, a message loop will get a stack overflow error, as you say. However, its cause can be hard to triage and diagnose. If can we write a system to anticipate these types of problems and flag them (e.g. message queues are not getting shorter as plug-ins process them, maximum message queue lengths) it can help not only prevent bugs but give developers a better impression of the system.

True. Though the only way I can think of off the top of my head to mark these is a trace or warn message on every send that includes the send method more than once in the stack, which is doable, but might be slow.

So we'll namespace the message IDs. That makes sense. This is probably already in your code and I did not see that.

Yep! And it is not, I have not yet begun to touch any message passing because I wanted to listen to feedback first. There are the beginnings of an event system, but Im considering it to still be distinct from "the" event system. Though I bet that will come back to bite me later. It may end up that its just two instances of an event management class.

Also I did some thinking. We could do message passing with optional returns? that gets us a "best of both" without increasing implementation complexity.

[Athanasius]

Btw, the initial impetus for an inter-plugin API was this discussion: https://discord.com/channels/164411426939600896/726419472772235306/755815747841818654

[inorton]

Nice job running things! really impressed at the turn around of new changes now!

One thing that has bugged me about plugins that I'd love to see changed is threadyness, I've had lots of problems where a plugin updating the main window would hog the main thread. It would be great if threads could each run on their own thread and could register attributes/functions for UI display. To encourage people to keep data processing and UI separate MVC style.

[A-UNDERSCORE-D]

Yeah it would be nice. Im poking it around in my head. Im not entirely sure we want to completely run plugins in their own thread (though @Athanasius might have a different opinion), that said, providing an event that promises to always be called synchronously to the tk thread would be nice. Possibly we can define what events promise being called that way, whereas others may be outside of the main thread.