Question - where does Clipper store all metric data?

[jacekwachowiak]

I tried to purge the data I already gathered, including the records, buckets and so on, but somehow I cannot find where it is stored - I want to be able to do that without stopping completely the cluster so that I have the data of the "new run" when I am testing a new model without having to specify that with every query in Prometheus/Grafana.

[rkooo567]

Hi, @jacekwachowiak

Based on the prometheus documentation here, https://prometheus.io/docs/prometheus/latest/storage/, metrics are stored at --storage.tsdb.path.

Our specified path will be "--storage.tsdb.path=/prometheus". So you should be able to find data from /prometheus path.

clipper/clipper_admin/clipper_admin/kubernetes/prom_deployment.yaml

Line 20 in 3c3957c

- "--storage.tsdb.path=/prometheus"

[jacekwachowiak]

That would mean that if I add the --web.enable-admin-api flag to the configuration file, I should be able to remove data when I want to, following https://www.shellhacks.com/prometheus-delete-time-series-metrics/. Maybe I have to try again. Has anyone tried anything similar?

[rkooo567]

@jacekwachowiak Could you find the data from the path? Also, what you mean is when you call the api after enabling the flag, you couldn't not delete them?

[jacekwachowiak]

I followed the article, to allow the external calls to Prometheus pod and then with a call to remove the data but I saw no change