The workload cluster flow is implemented by two controllers and one custom resource.
The ClusterAddon resource gets created by the ClusterAddonCreate controller for any Cluster resource that is applied.
The user never interacts with the ClusterAddon resource as it is created, updated, and deleted automatically.
It is updated by the ClusterAddon controller, which makes sure that all cluster addons are applied in the respective workload cluster.
The controller follows a simple pattern. When a cluster is created, it waits until the cluster is ready. If that is the case, it applies all objects from the ClusterAddon Helm Chart.
If a cluster is updated, it checks whether there has been an update of the cluster addons and only if that's the case, it applies the objects again. It also deletes objects that have been there in the previous version but are not there anymore.
Applying the objects has one additional step: we take the idea of the cluster-api-addon-provider-helm and add a few details about the Cluster and the ProviderCluster in there (https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm/blob/main/internal/value_substitutions.go).
This is necessary, because normal templating could not inject these values that are only available at runtime but that are very important to the resources that we apply as cluster addons.
As this controller relies on the release assets to be downloaded - as do other controllers that do not download anything themselves - there is one issue after a container restart that we have to solve:
If the container restarts, then everything that was stored in memory or without external volume in the container, will be lost. Therefore, a container restart requires to fetch from Github again.
This takes a bit of time, even if it is just one second. If a ClusterAddon reconciles within this one second, it willl realize though, that the desired file is not available yet. Instead of throwing an error, we can intelligently requeue again.
The same pattern is followed in all other controllers as well, if needed.
This controller also sets intelligent conditions into the status of the objects to make sure that the user can understand what is going on.