-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: cluster autoscaler #251
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #251 +/- ##
=======================================
Coverage 53.83% 53.83%
=======================================
Files 27 27
Lines 1566 1566
=======================================
Hits 843 843
Misses 673 673
Partials 50 50 ☔ View full report in Codecov by Sentry. |
1. Set up autoscaling environment variables | ||
```sh | ||
export CLUSTER_AUTOSCALER_VERSION=v1.29.0 | ||
export WORKER_MACHINE_MIN="\"1\"" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we move this string escaping into the template instead of the variable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think escaping should be needed, it's already quoted in the template
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can give this another shot, but envsubst
didn't play well with quoting numbers as strings in the templating. It was either stripping them too aggressively or not enough.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue here seems to be how clusterctl generate
generates templates in conjunction with envsubst
. For what I can see, it:
- Renders the templates and then validates the Kubernetes YAML
- Passes the generated YAML through
envsubst
envsubst
substitutes both ${var}
and "${var}"
so you need to explicitly specify the shell variable as a "string".
I've slightly modified the documentation commands to remove the escapes and also provided defaults for these values.
8e75985
to
1c70a57
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, I can't figure out a better workaround at the time for the scaling annotations 🤔
Adds a new cluster-autoscaler flavor that provides an autoscaling add-on for workload cluster nodes via [Cluster Autoscaler](https://www.github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#cluster-autoscaler). Due to constraints with the Kubernetes RBAC system (i.e. [roles cannot be subdivided beyond namespace-granularity](https://www.github.com/kubernetes/kubernetes/issues/56582)), the Cluster Autoscaler add-on is deployed on the management cluster to prevent leaking Cluster API data between workload clusters. Currently, the Cluster Autoscaler reuses the `${CLUSTER_NAME}-kubeconfig` Secret generated by the bootstrap provider to interact with the workload cluster. The kubeconfig contents must be stored in a key named `value`. Due to this, all Cluster Autoscaler actions in the workload cluster are performed as the `cluster-admin` role. See: https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling#autoscaler-running-in-management-cluster-using-service-account-credentials-with-separate-workload-cluster
* feat: add cluster autoscaler flavor Adds a new cluster-autoscaler flavor that provides an autoscaling add-on for workload cluster nodes via [Cluster Autoscaler](https://www.github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#cluster-autoscaler). Due to constraints with the Kubernetes RBAC system (i.e. [roles cannot be subdivided beyond namespace-granularity](https://www.github.com/kubernetes/kubernetes/issues/56582)), the Cluster Autoscaler add-on is deployed on the management cluster to prevent leaking Cluster API data between workload clusters. Currently, the Cluster Autoscaler reuses the `${CLUSTER_NAME}-kubeconfig` Secret generated by the bootstrap provider to interact with the workload cluster. The kubeconfig contents must be stored in a key named `value`. Due to this, all Cluster Autoscaler actions in the workload cluster are performed as the `cluster-admin` role. See: https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling#autoscaler-running-in-management-cluster-using-service-account-credentials-with-separate-workload-cluster * docs: add cluster autoscaler flavor
* feat: add cluster autoscaler flavor Adds a new cluster-autoscaler flavor that provides an autoscaling add-on for workload cluster nodes via [Cluster Autoscaler](https://www.github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#cluster-autoscaler). Due to constraints with the Kubernetes RBAC system (i.e. [roles cannot be subdivided beyond namespace-granularity](https://www.github.com/kubernetes/kubernetes/issues/56582)), the Cluster Autoscaler add-on is deployed on the management cluster to prevent leaking Cluster API data between workload clusters. Currently, the Cluster Autoscaler reuses the `${CLUSTER_NAME}-kubeconfig` Secret generated by the bootstrap provider to interact with the workload cluster. The kubeconfig contents must be stored in a key named `value`. Due to this, all Cluster Autoscaler actions in the workload cluster are performed as the `cluster-admin` role. See: https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling#autoscaler-running-in-management-cluster-using-service-account-credentials-with-separate-workload-cluster * docs: add cluster autoscaler flavor
What type of PR is this?
/kind feature
What this PR does / why we need it:
Adds a new cluster-autoscaler flavor that provides an add-on to autoscale workload cluster nodes via Cluster Autoscaler.
Testing
Ready
, download the kubeconfig file and deploy any workload.$ KUBECONFIG=/tmp/${CLUSTER_NAME}-kubeconfig kubectl scale deployment/nginx --replicas 2
MachineSets
,MachineDeployments
, orMachinePools
to meet scheduling requirements.$ KUBECONFIG=/tmp/${CLUSTER_NAME}-kubeconfig kubectl delete deployment/nginx
MachineSets
,MachineDeployments
, orMachinePools
. You may need to restart the Cluster Autoscaler with the--scale-down-unneeded-time=1s
setting for a quicker reaction time.Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)
format, will close the issue(s) when PR gets merged):Fixes #
Special notes for your reviewer:
Due to constraints with the Kubernetes RBAC system (i.e. roles cannot be subdivided beyond namespace-granularity), the Cluster Autoscaler add-on is deployed on the management cluster to prevent leaking Cluster API data between workload clusters.
Currently, the Cluster Autoscaler reuses the
${CLUSTER_NAME}-kubeconfig
Secret generated by the bootstrap provider to interact with the workload cluster. The kubeconfig contents must be stored in a key namedvalue
. Due to this, all Cluster Autoscaler actions in the workload cluster are performed as thecluster-admin
role (and might be insecure idk 🙈).See: https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling#autoscaler-running-in-management-cluster-using-service-account-credentials-with-separate-workload-cluster
TODOs: