v5: Re-consider replace-on-changes behavior for secrets and config maps

[blampe]

See Levi's comment here #1568 (comment)

As far as I can tell, the default replace behavior for configmaps and secrets came about because at the time Pulumi didn't have a concept of replaceOnChanges and Kubernetes didn't have immutable secrets or configmaps yet.

Those two gaps made it really difficult to bump Deployments to pick up new config, so Pulumi took the opinion that these should always be immutable. Hence deployments would always pick up new config/secrets because they would always be replaced.

Meanwhile, Pulumi has introduced replaceOnChanges and immutable secrets/cms have been stable since k8s 1.21. Given that, it would make sense for Pulumi to relax its opinion here and let the user decide the replacement behavior of these resources.

Related:

• Changing a secret's data always results in a replace insted of an update #1772