prometheus-node-exporter containers arguments not compatible with Azure AKS with ACI virtual nodes (scaleout)

[tpcgold]

There seems to be a problem on how the node exporter arguments are passed (without command)
Which does lead to a Problem deploying the k8s-monitoring chart to AKS with ACI scaleout enabled.

ACI does not support providing args without specifying the command. Please supply both command and args to the pod spec.

The problem seems to be that there is just:

  containers:
    - args:
        - '--path.procfs=/host/proc'
        - '--path.sysfs=/host/sys'
        - '--path.rootfs=/host/root'
        - '--path.udev.data=/host/root/run/udev/data'
        - '--web.listen-address=[$(HOST_IP)]:9100'

without a command.

Full Pod settings look like the following:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'
    k8s.grafana.com/logs.job: integrations/node_exporter
  creationTimestamp: '2024-09-25T07:19:17Z'
  generateName: grafana-k8s-monitoring-prometheus-node-exporter-
  labels:
    app.kubernetes.io/component: metrics
    app.kubernetes.io/instance: grafana-k8s-monitoring
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: prometheus-node-exporter
    app.kubernetes.io/part-of: prometheus-node-exporter
    app.kubernetes.io/version: 1.8.2
    controller-revision-hash: 6c78cfc98d
    helm.sh/chart: prometheus-node-exporter-4.39.0
    pod-template-generation: '1'
    release: grafana-k8s-monitoring
  name: grafana-k8s-monitoring-prometheus-node-exporter-qldcg
  namespace: default
  ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: DaemonSet
      name: grafana-k8s-monitoring-prometheus-node-exporter
      uid: 9076e57e-c7bc-4ee9-9704-6ee93c1024bb
  resourceVersion: '20317042'
  uid: 137cbc36-f6d4-4bc0-9fa8-5ebf21ad6d0b
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchFields:
              - key: metadata.name
                operator: In
                values:
                  - virtual-node-aci-linux
  automountServiceAccountToken: false
  containers:
    - args:
        - '--path.procfs=/host/proc'
        - '--path.sysfs=/host/sys'
        - '--path.rootfs=/host/root'
        - '--path.udev.data=/host/root/run/udev/data'
        - '--web.listen-address=[$(HOST_IP)]:9100'
      env:
        - name: HOST_IP
          value: 0.0.0.0
      image: quay.io/prometheus/node-exporter:v1.8.2
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 3
        httpGet:
          path: /
          port: 9100
          scheme: HTTP
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      name: node-exporter
      ports:
        - containerPort: 9100
          hostPort: 9100
          name: metrics
          protocol: TCP
      readinessProbe:
        failureThreshold: 3
        httpGet:
          path: /
          port: 9100
          scheme: HTTP
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      resources: {}
      securityContext:
        readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /host/proc
          name: proc
          readOnly: true
        - mountPath: /host/sys
          name: sys
          readOnly: true
        - mountPath: /host/root
          mountPropagation: HostToContainer
          name: root
          readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostNetwork: true
  hostPID: true
  nodeName: virtual-node-aci-linux
  nodeSelector:
    kubernetes.io/os: linux
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 65534
    runAsGroup: 65534
    runAsNonRoot: true
    runAsUser: 65534
  serviceAccount: grafana-k8s-monitoring-prometheus-node-exporter
  serviceAccountName: grafana-k8s-monitoring-prometheus-node-exporter
  terminationGracePeriodSeconds: 30
  tolerations:
    - effect: NoSchedule
      operator: Exists
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/disk-pressure
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/memory-pressure
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/pid-pressure
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/unschedulable
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/network-unavailable
      operator: Exists
  volumes:
    - hostPath:
        path: /proc
        type: ''
      name: proc
    - hostPath:
        path: /sys
        type: ''
      name: sys
    - hostPath:
        path: /
        type: ''
      name: root
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2024-09-25T07:19:17Z'
      status: 'True'
      type: PodScheduled
  message: >-
    ACI does not support providing args without specifying the command. Please
    supply both command and args to the pod spec.
  phase: Pending
  qosClass: BestEffort
  reason: ProviderFailed

Probably this should be an anti-affinity like with EKS.