Exposing custom indices and properties in Prometheus from OpenSearch

[ccamacho]

This is an usage question.

I have in my OpenSearch instance some indices and properties I will like to have exported in Prometheus. Is it possible to extend the current metrics that are exposed by default?

Thanks!

[lukas-vlcek]

Hi,

there are some plugin configuration options available. If you are interested in metrics from specific indices only then indices_filter might help.

As for properties this is not that flexible right now. Which properties would you be interested in?

Regards,
Lukáš

[ccamacho]

Thanks for the reply!

For example, lets add some random data in OpenSearch

    index_name = 'cpu_performance'
    index_body = {
        'settings': {
            'index': {
                'number_of_shards': 4
            }
        },
        "mappings": {
            "properties": {
                "cpu_idle_percentage": {
                    "type": "float"
                }
            }
        }
    }
    client.indices.create(index=index_name, body=index_body)
    
    for i in range(1, 101):  # Generate 100 data points
        timestamp = datetime.now() - timedelta(minutes=i)
        cpu_idle_percentage = round(random.uniform(0, 100), 2)
        document = {
            'cpu_idle_percentage': cpu_idle_percentage,
            '@timestamp': timestamp.strftime('%Y-%m-%dT%H:%M:%S.%fZ')
        }
        client.index(
            index=index_name,
            body=document,
            refresh=True
        )

once prometheus-exporter-plugin-for-opensearch is configured, and:

user@a5a0474dfa3f:~$ curl http://localhost:9200/_prometheus/metrics?pretty=true | grep cpu_performance | more
opensearch_index_indexing_delete_time_seconds{cluster="mytestcluster",index="cpu_performance",context="primaries",} 0.0
opensearch_index_indexing_delete_time_seconds{cluster="mytestcluster",index="cpu_performance",context="total",} 0.0
opensearch_index_suggest_count{cluster="mytestcluster",index="cpu_performance",context="primaries",} 0.0
opensearch_index_suggest_count{cluster="mytestcluster",index="cpu_performance",context="total",} 0.0
opensearch_index_status{cluster="mytestcluster",index="cpu_performance",} 1.0
opensearch_index_get_time_seconds{cluster="mytestcluster",index="cpu_performance",context="primaries",} 0.0

and my target is up as expected

[opensearch (1/1 up)](http://127.0.0.1:9090/targets?search=#pool-opensearch)
Endpoint	State	Labels	Last Scrape	Scrape Duration	Error
[http://localhost:9200/_prometheus/metrics](http://a5a0474dfa3f:9200/_prometheus/metrics)	UP	
instance="localhost:9200"job="opensearch"
10.251s ago	
12.707ms

Now in Grafana I can see several metrics related to the state of the OpenSearch instance, but not the metrics I created before.

My opensearch.yml has

prometheus.metric_name.prefix: "opensearch_"
prometheus.nodes.filter: "_all"
prometheus.indices_filter.selected_indices: "*"

Id like to see from my cpu_performance index, the cpu_idle_percentage and the @timestamp values.