Maybe a bug with kube state metrics kube_node_labels reporting

[gete76]

I'm not sure if this is intended behavior for this specific metric. For our metrics reporting coming from kube state metrics, unlike other metrics like kube_node_info whose node dimension reports the exporting nodes name, the node dimension in the kube_node_labels metric is reported as the node that the kube state metrics service is running on. This causes the metric to aggregate on the node dimension, instead of reporting per node, it is reporting only a single metric stream or a few (depending on what labels you allow, it will aggregate on them). I managed to get the proper node name info or aggregation by adding hostname label to the allowed labels list but it seems like this node dimension should report the actual node the labels are on. This also prevents me from joining other metrics with it on the node dimension.

Operator chart version 4.4.0
k8s v1.28.12-eks
kube-state-metrics v2.7.0
node-exporter v1.3.1
prometheus-operator v0.59.2
sumologic-otel-collector 0.92.0-sumo-0