Support Webhooks for Kamaji Events

[JonnyBDev]

We would love to see support for Webhooks in Kamaji for specific / all events from the operator. As of now the operator is doing a lot of actions but from an external perspective you can't really tell the status of these actions. Some examples for these actions are:

• Creating a new TenantControlPlane
• Upgrading / Patching an existing TenantControlPlane
• Deleting an existing TenantControlPlane

Unfortunately we only know the status of the current action by polling the TenantControlPlane. And even with polling, we can not be 100% sure that the action has been executed due to potential delays in the operator. The only option we have right now is to delete the TenantControlPlane. After the TenantControlPlane is gone, we should consider it deleted and not available anymore. But that's just wrong. A deletion of a TenantControlPlane triggers a reconciler cycle which will delete all resources in the given namespace that are connected to the TenantControlPlane. If something goes wrong here or the operator is still working on other actions, we can not safely terminate the namespace due to finalizers. Not being able to delete resources due to running finalizers is not a good way to automated workflows like a deployment of a Kubernetes cluster. The risk of artifacts in a cluster with a high number of TCPs is pretty high in that scenario.

One example would be the Benchmark with 100 TCPs. You stated that with 100 TCPs it would take around 7.5 minutes for ONE reconciliation cycle (4.5s / TCP). Let us draw an example usecase around this.

Imagine we would like to upgrade all our 100 TCPs to the next Kubernetes version. We would fire 100 API-Calls to k8s at once to modify the TenantControlPlane with the latest number. We would have to permanently poll all TCPs to get the current status. As you mentioned in your Benchmark, next up would be a test with 1000 TCPs which would, if we keep going with a linear increase, set the reconciliation cycle at 75 minutes. This would make mass updating clusters very complex and not efficient.

We would like to see something implemented that would notify us instead of polling all TCPs. As described above all actions the operator executes should end in a configurable notification webhook. This way MSPs could just start like 1000 TCP-upgrades and get notified with the latest status after the upgrade happened. Another questionable thing would be the current state of the cluster. Is it "Ready" or "NotReady"? Is it experiencing problems? We would have to poll each TCP for the current status basically every 5 seconds instead of getting notified in realtime right after an event occured. Webhooks are the state of art technique to communicate events. You may say that this would be out-of-scope for Kamaji because you are just providing the tooling and not the platform around this. I think that's the wrong approach for a software like this. You are obviously providing a great tool but if you want Kamaji to be used by big MSPs, you should give them the tools to build an automation platform around this with features like Webhooks.

There are a lot of advantages in Webhooks vs polling. Just to name a few:

• Less resource hungry
• Better scalability (important for large MSPs)
• Faster reaction times for important events (e.g. TCP not ready)
• Less network traffic

One example for a integration in the tenentcontrolplane_controller.go would be here. Beside logging the request in the container, just add a HTTP call to a configurable external webhook url. My approach would be that I'd notify for each event the operator receives. Just send a "Starting to delete TCP" (START) and "TCP started successfully"(END) webhook and everyone who uses and automates your software will be forever grateful.

Thanks for reading this :)

[avorima]

Webhook integrations are not very common in Kubernetes controllers. I think the main reason is that Kubernetes already has good alternatives.
The watch API can be used to observed state or state transitions of resources. It's more efficient than polling and there are ways to further improve performance using caching.
For cleanup there are finalizers on objects that block deletion until they are removed. Lots of controllers make use of them which is why it's possible to delete a namespace that still has resources in it and eventually it will be deleted. In between these two points in time the state will indicate an error, but it is transitive. This ties into the eventual consistency model of Kubernetes.

[JonnyBDev]

Hello @avorima,

first of all thank you for your answer and your explanation. I'd like to discuss some points that are really important for us as MSP.

You are right - webhook integrations are not very common in Kubernetes Operators. Often those operators rely on their users to utilize some API from Kubernetes to get the current status of the deployed objects. I can completely understand this pattern. I'd like you to explain our side here why a webhook integration for events would help us. We would like to host several hundreds of TCPs for our customers. We are already building a platform that will deploy the TCPs for us. With like 500 TCPs we would have to have a pretty big long running api call to get all this information and we'd have to compute it every time. Modern applications are using webhooks as an easy way to emit events to external applications to process them there. This would help us a lot! We would then only call Kamaji for the deployment / update / upgrade / deletion and we'd get notified after a sucessful event. We as a managed service provider shouldn't have to poll all the time to get the current status. For the first let's say 50 TCPs that's a valid point and you obviously solved this issue. But we need this kind of behaviour for hundreds of TCPs. That's a whole other level.

For the watch api i can not 100% agree with you. Yes, we could use it as mentioned above but this would have limits. Additionally "using caching to further improve performance" would not be helpful. We'd like to get the latest state of a TCP not the one from 10 minutes ago.

For the finalizer part - I partly agree with you. In a perfect working environment that's working fine - I agree. But with a unavailable datastore artifacts aren't deleted completely. Had this while testing PosteSQL. We couldn't connect to the database because some certificate issue. We created a TCP and deleted it. Obviously a lot of errors in the operator but after the deletion of the TCP and the namespace still a ton of logs about the already deleted TCP. The reason was that some artifacts of the TCP weren't deleted.

To summarize things. We are a managed service provider and we are currently providing Kubernetes clusters for our customers. We have to be informed for every event that occurs so that we can act accordingly. That's why sending out the logs in the operator as event to a webhook would MASSIVELY help us here. I understand that you have solved this problem for Kamaji and that an operator may not use pattern like webhooks. But for this usecase I don't see Kamaji only as an operator. I see it as THE brain of our managed Kubernetes clusters. And if the brain can not communicate events with current standards in realtime for us to act immediately, that's not good for us.

[JonnyBDev]

May I ask if there's been any process internally about this topic? I don't expect an answer right away but some kind of reaction would be nice that we know in which direction this discussion is heading.

As described above we are 95% done in implementing Kamaji in our service but some key components like TCP status w/o polling is still missing for us to actually use is as a Managed Service.

[prometherion]

I'd say the answer given back from @avorima is definitely on topic and highlights why webhooks are wrong here, sharing my thoughts here too.

If a platform is relying on webhooks, it means such information is vital for its state record, such as creation, update, and delete: it means these events must not be missed since it would lead to an inconsistent track state. That would require handling events in a different manner as you suggested, such as doing bare HTTP calls for the following reasons:

1. the webhook receiver couldn't be ready to handle the request, requiring a backoff mechanism to retry
2. the webhook emitter, such as Kamaji, could have connectivity issues, requiring a back-off mechanism
3. with both cases, these calls should be tracked externally since if Kamaji is restarted during one of these windows, those events would get lost

To achieve this kind of functionality it means an additional store is required to store events, and executed calls, such as a Database keeping track of the queued and executed actions, and this is the first huge blocker here: which database should be picked? There are so many alternatives out there and whatever stack we pick, would be considered opinionated since a stack requires expertise in operating a specific database, with its lifecycle, etc. (you can imagine the battles between why picking up PostgreSQL instead of MySQL, or the requests you should provide a Data Layer to get it abstracted and applicable to any RDBMS, or The Database instance must be installed, maintained, and operator through the Helm Chart). In a nutshell, it would increase the stack required to run Kamaji, considering the current one which is pretty straightforward and lightweight.

Kamaji is sticking to a specific, recognized, and priceless conceptpriceless that every Linux enthusiast knows: do one thing and do it well. We're glad to see Kamaji used by several organizations in several ways and being the foundation of bigger platforms, and I'll be arrogant here playing the seniority card here given the fact we operated huge platforms and designed Kamaji on top of our knowledge is very aware of designing a monolith. We gathered huge experience in doing open source prior with Capsule and Kamaji is adhering to the same principles, such as being a framework (or an Infrastructure SDK) to take advantage to build a higher abstraction for Kubernetes clusters management, based on a concept of top down rather than bottom up in the infrastructure stack.

As I said, we're pleased to see Kamaji used broadly by commercial actors making their own business. At the same time, we cannot solve specific business needs: our mission is to keep Kamaji extensible as much as possible and keep the development roadmap pleasant for our customers, fueling the project's development with cash, which we all require to survive.

If this feature is missing from your platform, I'd say I have to congratulate you since it's an absolute achievement since a perfect fit (or 100% criteria match) by just glueing open source projects is very challenging. We'll be happy to accept contributions in a different repository to propagate webhooks to listeners to address your platform needs, as well as get hired to develop this sort of addon as we did with Capsule in the past. Development is hard, time and money-consuming, since it's business for you (such as generating revenue for your company or the company you're working with) sponsoring this development sounds feasible to me. As previously said, Kamaji is Open Source but only maintained by CLASTIX: we have limited resources, and we give preference to our customers which are requiring a different set of features, and unfortunately, webhooks is not one of these.

tl;dr; happy to receive contributions, as well as get engaged in the development with a SoW enrollment.

If you're expecting custom developments for free for your own business, I'm sorry, we're not open to that, and wishing you all the best in picking up another solution.

[bsctl]

@JonnyBDev Thank you for giving Kamaji a try. Happy to see you’re finding it valuable for your project. As highlighted by @prometherion, implementing webhooks in Kamaji for notifications about status is out of scope and frankly not the best tech choice. However we totally understand your business need for status notification and, as it makes much more sense, we’re open to help you in developing a tool to achieve it. Feel free to reach us in private so we can jump on the discussion. Thank you.