Unable to remove deletion protection on CloudSQL instance

[jond3k]

What happened?

Once removing a CloudSQL instance with deletionProtection: true set has failed it will not cleanup resources even once the flag is removed (The finalizer is not retried)

How can we reproduce it?

1. Create a DatabaseInstance with deletionProtection: true (this flag is unnecessary because it's the default value)

> echo "apiVersion: sql.gcp.jet.crossplane.io/v1alpha2        
kind: DatabaseInstance
metadata:
  name: my-db-with-dp
spec:
  forProvider:
    region: europe-west3
    deletionProtection: true
    settings:
    - tier: db-custom-1-3840
    databaseVersion: MYSQL_8_0" | kubectl apply -f -

2. Attempt to delete the instance

> kubectl delete DatabaseInstance my-db-with-dp

databaseinstance.sql.gcp.jet.crossplane.io "my-db-with-dp" deleted

3. This will -- correctly -- fail, as seen in the status

> kubectl get DatabaseInstance my-db-with-dp -o yaml
apiVersion: sql.gcp.jet.crossplane.io/v1alpha2
kind: DatabaseInstance
metadata:
  ...
  finalizers:
  - finalizer.managedresource.crossplane.io
  name: my-db-with-dp
spec:
  forProvider:
    deletionProtection: true
    ..
status:
  ...
  conditions:
    ...
  - lastTransitionTime: "2022-08-30T15:07:22Z"
    message: 'destroy failed: Error, failed to delete instance because deletion_protection
      is set to true. Set it to false to proceed with instance deletion: '
    reason: DestroyFailure
    status: "False"
    type: LastAsyncOperation

4. Remove the deletionProtection flag

kubectl patch DatabaseInstance my-db-with-dp --type='json' -p='[{"op": "replace", "path": "/spec/forProvider/deletionProtection", "value":false}]'

databaseinstance.sql.gcp.jet.crossplane.io/my-db-with-dp patched

5. You can return later and it's still not deleted, the status hasn't been updated. Effectively it is not retrying to delete.

6. Attempting to 'undelete' the object by removing deletionTimestamp and retrying the kubectl delete also fails. Only removing the finalizer and deleting manually works.

What environment did it happen in?

Crossplane: v1.9.0
Cloud provider: crossplane/provider-jet-gcp:v0.3.0-rc.35.g40c122f
Client Version: v1.24.1
Server Version: v1.22.10-gke.600
Distro: GKE Autopilot