Exporting only specific metrics from detailed index metrics

[hagaram]

Hi, I know this might not be a standard use case, but I will try to ask anyways.

When exporting detailed metrics, the size of a scrape is pretty large in our case and of course grows with more indices.
We don't want to "throw" the metrics away on the prometheus/vicotriametrics level, as this doesnt solve the issue with very large metrics (even hitting some thesholds in prometheus resulting in errors) .

Isn't it somehow possible to export detailed metrics, but lets say only one single metric, example: doc count per index ?

Regards,
Michal

[lukas-vlcek]

Hi @hagaram,

I know that we need to make the metric pulling more flexible because there are limitations on metric stores and sooner or later these are hit as the OpenSearch cluster grows. So the idea that all metrics can be pulled within a single request/response does not fit a lot of practical use cases. I was playing with an idea to make it possible to pull various metric sets at various intervals because "Not all metrics are born equal".

In a short term I think there should be a way how to help you. There is an indices filter config option that you can use to get info about specific indices only. At this point this filter is config level only but it shouldn't be hard to allow for this filter to be overridden by URL parameter.

What that means is that then you could define several targets for scraping in Prometheus and each would be using a different mask to include a different "slice of indices". For instance you could defined three targets each using distinct index name mask like: ["product-*", "catalog-*", "system-*"].

Then the amount of metrics that is pulled from OpenSearch would not be matter of configuration of the plugin but configuration in Prometheus.

WDYT?