Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[prometheus-blackbox-exporter] Problem with Metric Relabelings for Self Monitor. #4834

Open
owantula opened this issue Sep 4, 2024 · 1 comment · May be fixed by #4882
Open

[prometheus-blackbox-exporter] Problem with Metric Relabelings for Self Monitor. #4834

owantula opened this issue Sep 4, 2024 · 1 comment · May be fixed by #4882
Labels
bug Something isn't working

Comments

@owantula
Copy link

owantula commented Sep 4, 2024

Describe the bug a clear and concise description of what the bug is.

There is a problem with the additionalMetricsRelabels for Service selfMonitor.

When using config-reloader, then inside the template for it there isn't even an option for it to be loaded and it's ignored. Even though it can be added inside values.

It misses the metricRelabelings: section in here:

  - path: {{ .Values.configReloader.serviceMonitor.selfMonitor.path }}
    interval: {{ .Values.configReloader.serviceMonitor.selfMonitor.interval }}
    scrapeTimeout: {{ .Values.configReloader.serviceMonitor.selfMonitor.scrapeTimeout }}
    scheme: {{ .Values.configReloader.serviceMonitor.selfMonitor.scheme }}
    {{- if .Values.configReloader.serviceMonitor.selfMonitor.tlsConfig }}
    tlsConfig:
      {{- toYaml .Values.configReloader.serviceMonitor.selfMonitor.tlsConfig | nindent 6 }}
    {{- end }}
    {{- if .Values.configReloader.serviceMonitor.selfMonitor.additionalRelabeling }}
    relabelings:
      {{- toYaml .Values.configReloader.serviceMonitor.selfMonitor.additionalRelabeling | indent 6 }}
    {{- end }}
  {{- end }}
  jobLabel: "{{ .Release.Name }

For the selfMonitor for the blackbox-exporter itself, there is a section for additionalMetrics, which adds them

Which doesn't work because this expects an array, but in values we expect a map.

It seems the easiest fix is either, to change the expected type for additionalMetricsRelabels to an array inside the values. Or use the working template for serviceMonitor metricRellabeling and use that one for selfServiceMonitor.

What's your helm version?

version.BuildInfo{Version:"v3.15.4", GitCommit:"fa9efb07d9d8debbb4306d72af76a383895aa8c4", GitTreeState:"clean", GoVersion:"go1.22.6"}

What's your kubectl version?

Client Version: v1.31.0

Which chart?

prometheus-blackbox-exporter

What's the chart version?

9.0.0

What happened?

We bumped the version from 8.16.0 to 9.0.0 which activated the option for AdditionalMetricRelabeling. Since we had it in a map, but it started expecting array our deployment failed and we had to figure out what happened.

The error we got was

error validating data: ValidationError(ServiceMonitor.spec.endpoints[0].metricRelabelings): invalid type for com.coreos.monitoring.v1.ServiceMonitor.spec.endpoints.metricRelabelings: got "map", expected "array"

When we tried to use array we got:

coalesce.go:286: warning: cannot overwrite table with non table for blackbox-exporter.prometheus-blackbox-exporter.serviceMonitor.selfMonitor.additionalMetricsRelabels (map[])

What you expected to happen?

Nothing, everything to keep "working" as before. But well we found out some things did not ever work.

How to reproduce it?

Enable additionalMetricsRelabels, inside selfServiceMonitor.

Enter the changed values of values.yaml?

prometheus-blackbox-exporter:
  serviceMonitor:
    enabled: true
    selfMonitor:
      enabled: true
      labels:
        app: blackbox-exporter
      additionalMetricsRelabels:
        team: observability
    defaults:
      labels:
        app: blackbox-exporter
      additionalMetricsRelabels:
        team: observability
  configReloader:
    enabled: true
    serviceMonitor:
      selfMonitor:
        additionalMetricsRelabels:
          team: observability


Enter the command that you execute and failing/misfunctioning.

Used our script, we use for templating with our ArgoCD. But it was just helm template with few parameters.

helm template --api-versions=${KUBE_API_VERSIONS} --kube-version=${KUBE_VERSION} ${ARGOCD_ENV_RELEASE_NAME:-wandera} ${NAMESPACE_PARAM} ${ARGOCD_ENV_HELM_ARGS} $(printf " -f %s.yaml " "${VALUES[@]}") . > all.yaml

Anything else we need to know?

No response

@owantula owantula added the bug Something isn't working label Sep 4, 2024
@owantula owantula changed the title [prometheus-blackbox-exporter] Issue Title [prometheus-blackbox-exporter] Problem with Metric Relabelings for Self Monitor. Sep 12, 2024
@Sheikh-Abubaker
Copy link
Contributor

I'll look into it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants