After adding a tag to a subnet, it can't be removed

[fiunchinho]

/kind bug

What steps did you take and what happened:
Using managed VPC and subnets, I wanted to add tags to the subnets that CAPA would create. For that I set the desired tags in the AWSCluster.spec.network.subnets[].tags. The tags were added successfully to the created subnets. But If I delete any of the tags, they won't ever be removed.
This is because CAPA network reconciler goes to fetch the existing subnets on AWS, and copies the details to the subnets in the AWSCluster CR. This means that whatever tags already exist on AWS are the tags that will be used during the reconciliation.
Removing the tag directly from AWS, allows me to delete the tag. But tags need to be deleted the same way they were added.

What did you expect to happen:
Deleting tags from subnets defined in AWSCluster CR would actually remove the tags on the AWS Subnet resource.

Anything else you would like to add:
It looks like the source of truth is the AWS Subnet resource, rather than the subnet spec in the AWSCluster. Is this by design or a mistake?

Environment:

• Cluster-api-provider-aws version: 2.5.2
• Kubernetes version: (use kubectl version): 1.28
• OS (e.g. from /etc/os-release):

[dlipovetsky]

/triage accepted
/priority important-longterm

[fiunchinho]

I think the main problem is that we are copying the existing subnet (meaning the AWS Subnet resource) into the subnet defined in the CR. This means all tags present on the AWS Subnet will be copied over the AWSCluster CR, which will be reconciled. Why do we need to copy the whole object? Maybe @richardcase knows. Would it be enough copying needed/required fields i.e. subnet id, route table id, etc?