[kube-prometheus-stack] : Support for GKE Autopilot

[nistiwar]

Is your feature request related to a problem ?

I am trying to install kube-prometheus-stack helm chart on a GKE Autopilot cluster with Allowlisted workloads, not successful. Error details:
$ helm install kube-prometheus-stack . -n monitoring Error: INSTALLATION FAILED: 6 errors occurred: * services is forbidden: User "nishith*****.com" cannot create resource "services" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied * services is forbidden: User "nishith*****.com" cannot create resource "services" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied * services is forbidden: User "nishith*****.com" cannot create resource "services" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied * services is forbidden: User "nishith*****.com" cannot create resource "services" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied * services is forbidden: User "nishith*****.com" cannot create resource "services" in API group "" in the namespace "kube-system": GKEAutopilot authz: the namespace "kube-system" is managed and the request's verb "create" is denied * admission webhook "gkepolicy.common-webhooks.networking.gke.io" denied the request: GKE Policy Controller rejected the request because it violates one or more policies: {"[denied by autogke-disallow-hostnamespaces]":["enabling hostPID is not allowed in Autopilot. Requested by user: 'nishith*************.com', groups: 'system:authenticated'.","enabling hostNetwork is not allowed in Autopilot. Requested by user: 'nishith*************.com', groups: 'system:authenticated'."],"[denied by autogke-no-host-port]":["container node-exporter specifies a host port; disallowed in Autopilot. Requested by user: 'nishith*************.com', groups: 'system:authenticated'."],"[denied by gec-hostpath]":["hostPath volume proc used in container node-exporter uses path /proc in read mode which is not allowed. Allowed read path prefixes for hostPath volumes are: [/dev/hugepages /dev/infiniband /dev/vfio /dev/char /sys/devices]. Requested by user: 'nishith*************.com', groups: 'system:authenticated'.","hostPath volume sys used in container node-exporter uses path /sys in read mode which is not allowed. Allowed read path prefixes for hostPath volumes are: [/dev/hugepages /dev/infiniband /dev/vfio /dev/char /sys/devices]. Requested by user: 'nishith*************.com', groups: 'system:authenticated'.","hostPath volume root used in container node-exporter uses path / in read mode which is not allowed. Allowed read path prefixes for hostPath volumes are: [/dev/hugepages /dev/infiniband /dev/vfio /dev/char /sys/devices]. Requested by user: 'nishith*************.com', groups: 'system:authenticated'."]}
node-exporter-ds.txt
Allowlistedworkloads.txt

Describe the solution you'd like.

There should be-
1- Documented way to allow node-exporter to be able to deploy on a GKE Autopilot cluster
2- The services created in kube-system namespace should be avoided and still we should be able to scrape components like scheduler/kubelet etc.
3- There should be a node label to metrics collected by node-exporter, along with instance

Describe alternatives you've considered.

Have tried:
1- Disabling scheduler,dns,kubelet, etc components.
2- Tweaking Node-exporter to comply with GKE Autopilot, i.e. remove permissions from node-exporter which are restricted by GKE Autopilot.
3- Tried to use Allowlistedworkload CRD to escalate priveleges for node-exporter.

1 and 2 worked, but left us with limited metrics and most of the panels not working in dashboards. 3, should allow node-exporter to run with elevated privileges in GKE Autopilot, but I was not successful implementing it.

Additional context.

No response