Pluggable Executors - Phase II

[rossjones]

Edit: We are now retargetting this work to https://www.notion.so/expanso/Executors-a3c19862cf3b496589e7537cb2200e94 - a lot of the tickets are still valid, but with different content.

We currently have two pluggable executors that can run as separate processes (docker and wasm), and can also run embedded in the main process. Whilst we do have the pluggable executors and they can be configured to run separately, there is further work required to iterate on them. This work will enable future external plugins, and it will be required to enable the Python executor.

Tasks

Beta Give feedback

1. https://github.com/bacalhau-project/expanso-planning/issues/118
2. https://github.com/bacalhau-project/expanso-planning/issues/132
3. Plugin configuration for compute node #3947
   comp/executor/plugins +1
4. Build protobufs in compute/executors #3913
   comp/executor/plugins +1
5. Add plugin supervisor #3908
   comp/executor/plugins +1
6. Add service to plugin supervisor #3909
   comp/executor/plugins +1
7. Plugin configuration for plugin #3921
   comp/executor/plugins +1
8. Investigate configurable cli params for CLI plugins #3916
   comp/executor/plugins +1
9. Refactor existing executors to /plugins folder and reduce dependencies #3920
   comp/executor/plugins +1
10. Fix build process for plugins + how we ship them #3919
    comp/executor/plugins +1
11. Fix protobufs to use realistic types and fix executors #3918
    comp/executor/plugins +1
12. Implement supervisor heartbeat #3907
    comp/executor/plugins +1
13. Implement TestPlugin in line with IDL for testing #3906
    comp/executor/plugins +1
14. Build 'environment' in compute node for plugin #3912
    comp/executor/plugins +1
15. Integrate plugin's logging #3911
    comp/executor/plugins +1
16. Update logserver to use new files + library #3910
    comp/executor/plugins +1

[rossjones]

/cc @simonwo

[simonwo]

@rossjones Is this still the right list given your doc?

[rossjones]

No, I was supposed to have reworked them yesterday, but will do so today.

[wdbaruni]

deprioritizing all work related to pluggable executors for now