S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time

[brandond]

K3s tracking issue for:

• [BUG] etcd snapshot count reported in rancher is incorrect when taking manual snapshot with s3 enabled on rke2 clusters, but is correct in s3 rancher/rancher#47339

This is extremely difficult to trigger outside of Rancher. When the user triggers an etcd snapshot via Rancher, all etcd nodes simultaneously save snapshots and attempt to reconcile their local snapshots, and the snapshots they see on S3, with ETCDSnapshotFile resources. Since there is a slight time gap in between when objects are listed from S3, and when ETCDSnapshotFiles resources are listed, not all cluster members will have a consistent view of objects in both stores, and may end up deleting ETCDSnapshotFile resources for snapshots that other nodes just finished uploading. The resources will be recreated next time S3 is reconciled (the next time snapshots are saved/deleted/pruned) but this does cause a temporary mismatch between what shows up in kubectl get etcdsnapshotfile and kubectl get configmap -n kube-system k3s-etcd-snapshots and what shows up in k3s etcd-snapshot ls (and what's actually in S3).