[prometheus] Unclear implications around running as a StatefulSet

[SeanKilleen]

Using a blank issue because it doesn't rise to the level of a bug but is something that a PR could fix and that I'd be happy to contribute back once I understand the approach.

Challenge

It is unclear to an end user (or at least a less intimately familiar end user) what the implications are around running this chart with 2+ replicas as a StatefulSet.

The replicas both spin up their own PVCs of equal size unless a pre-created PVC is specified.

Questions naturally arising from this:

• If using a StatefulSet, should I specify one PVC so that both replicas are using the PVC? (or would that break things?)
• Does this naturally create a cluster where those Prometheus nodes are communicating? Or are they running independently of one another?
• Can I still set e.g. my Grafana dashboard to pull from the service in front of these nodes (or from one node specifically)? Or do I need to modify this approach to consider an additional data source?

Suggestion

Once these implications are understood here, I can update the values.yaml etc. to reflect the shared understanding via a PR and close this out.

Impact

I think this will make it easier for folks newer to Prometheus who want to build a little resiliency and are looking to understand the default behavior.