Reliable Orchestration

[wdbaruni]

The Problem

Today we have many situations where the requester node thinks a compute node is still running job while it isn't, and a compute node is till running a job where it shouldn't. A major reason for this is the way we communicate between the requester and the compute node where we only rely on live communication and lack periodic checks.

As an example, when a compute node finishes execution, it communicates that to the requester by calling OnRunComplete on the requester node, but the requester might be down at that time, the compute node might be isolated, or the requester might face a transient failure and don't handle the requester properly (e.g. transient failure when writing to the job store).
A similar issue can happen when the user stops a job, and the requester forwards that to the compute node, but the compute node was unreachable at that time, which means the compute node will continue to run the job even though the user and requester state says otherwise.

The Proposal

Multiple changes are required to improve the communication path between the requester and compute nodes, including:

Tasks

Beta Give feedback

1. Design reliable orchestration #4134
   1 of 1
   +0
   
2. Core Event System #4261
   4 of 4
   comp/ncl type/epic +2
   
3. BoltDB Event Logging #4267
   4 of 4
   comp/ncl type/epic +2
   
4. Improved Logging #4283
   comp/ncl type/epic +2
   
5. Async Job Assignment #4272
   comp/ncl type/epic +2
   
6. State Synchronization #4276
   comp/ncl type/epic +2
   
7. Event Management #4280
   2 of 2
   comp/ncl type/epic +2
   
8. Failure Handling #4286
   comp/ncl type/epic +2
   

[wdbaruni]

Replacing the issue with a linear project