Improve alert aggregations for multiple clusters

[hamishforbes]

Many of the alerts are aggregating on metrics and throwing away the cluster label.

While this isn't the end of the world, pod names are probably going to be unique, but if you have many clusters trying to figure out which one kube-proxy-4jlm2 is running on is a pain.

Because these labels are currently just hardcoded into strings its also difficult to override them in your own jsonnet.

This PR adds extra labels to aggregations.
Only if multi cluster is enabled and extensible.

For example in my environment I also inject a dc label with the AWS region for all metrics as well as the cluster label.
So in my config I can do

_config+:: {
  showMultiCluster: true,
  clusterGroupLabels+: ['dc'],
}

and include that dc label in aggregations

[hamishforbes]

Hi, is there anything else I can do to move this along?
Any changes or tests required?
ATM i'm maintaining a fork for this which is a bit of a pita.

I did just find #524, which I missed when making this PR but largely does the same thing, that's been hanging around since last year.
Is there just no desire to support multiple clusters in alerting?

[paulfantom]

Could this be using clusterLabel ? I don't see why we would need another variable for this.

Same question for all cases with clusterGroupLabelsStr

[hamishforbes]

The default value for clusterGroupLabels is clusterLabel, so in the simple case they are the same thing.

But clusterLabel is used in other places (e.g. in dashboards as a selector like %(clusterLabel)s="$cluster") and so can only be a single label.

Using a different variable for grouping labels gives more flexibility and allows use cases like I described in the PR, where I ideally want to group by $clusterLabel, dc and not just the clusterLabel.