Multicluster support for k8s mixin alerts & rules

[vishiy]

showMultiCluster is useful in controlling multiCluster behavior for dashboards. But i dont see it used consistently across across all rules & alerts in k8s mixin. Is that a bug ? In multi cluster stores, this would be a problem . Anyone have any thoughts/guidance here ? Much appreciate your help.

[povilasv]

hey, could you give concrete examples?

[vishiy]

I see it in many places for rec.rules & alerting rules - ex;- below - so if i need to run this on a multi cluster store, it needs to account for cluster/resource dimension/label. I also see this when i look at core-dns or node mixins , which are defaulted by kube-stack along with k8s-mixin.

• "alert": "KubeCPUQuotaOvercommit"
  "annotations":
  "description": "Cluster has overcommitted CPU resource requests for Namespaces."
  "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuquotaovercommit"
  "summary": "Cluster has overcommitted CPU resource requests."
  "expr": |
  sum(kube_resourcequota{job="kube-state-metrics", type="hard", resource="cpu"})
  /
  sum(kube_node_status_allocatable{resource="cpu"})
  > 1.5
  "for": "5m"
  "labels":
  "severity": "warning"

[phillebaba]

@povilasv basically any rule or alert that does 1 to 1 merging of series based on the node ip has this problem.

histogram_quantile(0.5, sum(rate(kubelet_pleg_relist_duration_seconds_bucket[5m])) by (instance, le) * on(instance) group_left(node) kubelet_node_name{job="kubelet"})

The example above will have issues in an instance where two clusters have nodes with the same ip. In this case the value of instance would be the node ip. This query will fail without any additional labels to differentiate between the different timeseries.