[proposal] Reservation support binding to scheduled pods

[saintube]

What is your proposal:

Besides reserving the unallocated node resources for further pods, reservations can match and bind to the scheduled pods on nodes.

Why is this needed:

Currently, the reservation can allocate the free node resources and reserve them for further pods (i.e. owners). In some advanced use cases, we want to keep a certain quantity of reserved resources in the cluster to ensure the resources are deterministic to deliver to users, so we reserve resources with reservations. Firstly, some user pods are scheduled and bound with reservations, and more pods are scheduled without any reservation. Once the user pods bound with reservations terminate, there are pods lacking reservations and reservations unbound to any pod. If we keep the unbound reservations, cluster resources are wasted. Otherwise, we clean up the unbound reservations, then the reserved resources are insufficient and the user pods are not assured to schedule deterministically.

For this case, we need the ability to let the reservation bind to scheduled pods. So when the reservations become unbound from the terminating pods, we can bind the reservations to other scheduled pods to keep the quantity of the reserved resources.

It also can be an alternative to the pre-allocation mentioned in the reservation proposal. Reservation can first bind to a scheduled pod, and then the resources will be reserved after the pod terminates. It helps when the cluster resources are insufficient currently, but we want to pre-allocate resources for some pods to reschedule after the resources are ready.

Is there a suggested solution, if so, please add it:

This ability should be an option for a reservation object since not every reservation needs to bind scheduled pods.
It should support both unscheduled reservations and available reservations considering the pre-allocation scenarios.
The binding logic is recommended to implement in the koord-scheduler due to the consistency problem when new pods also could allocate the unbound reservations.