Load Balancer DNS resolution takes too long

[r4f4]

/kind bug

What steps did you take and what happened:

This issue was discovered by @jianlinliu, @gpei, and @yunjiang29.

As the load balancer reconciliation currently stands, CAPA tries to check for LB DNS name resolution right after the load balancer is created. The DNS name resolution success is necessary for CAPA to deem the LB as ready [1].

DNS name propagation can usually take a few minutes. If we try to resolve the name too soon, we might get:

$ nslookup -debug lb.notvalid.address.com
Server:		127.0.0.53
Address:	127.0.0.53#53

------------
    QUESTIONS:
	lb.notvalid.address.com, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  address.com
	origin = ns-93.awsdns-11.com
	mail addr = awsdns-hostmaster.amazon.com
	serial = 1
	refresh = 7200
	retry = 900
	expire = 1209600
	minimum = 86400
	ttl = 600
    ADDITIONAL RECORDS:
------------
** server can't find lb.notvalid.address.com: NXDOMAIN

and the name resolution will keep failing with the cached answer for the next TTL=600s=10min.

We have observed in some systems that the TTL for LB DNS in the us-gov-west-1 region could be as high as 900s.

$ dig jialiu-735f8ebd8bc9a14e.elb.us-gov-west-1.amazonaws.com; <<>> DiG 9.11.36-RedHat-9.11.36-5.el8_7.2 <<>> jialiu-735f8ebd8bc9a14e.elb.us-gov-west-1.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 27112
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
; COOKIE: 2caaa84d7bd004acf1e8a4aa667508daacf8365c3d922552 (good)
;; QUESTION SECTION:
;jialiu-735f8ebd8bc9a14e.elb.us-gov-west-1.amazonaws.com. IN A;; AUTHORITY SECTION:
elb.us-gov-west-1.amazonaws.com. 900 IN    SOA    ns-1151.awsdns-15.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 60;; Query time: 12 msec
;; SERVER: 10.11.5.160#53(10.11.5.160)
;; WHEN: Fri Jun 21 01:00:10 EDT 2024
;; MSG SIZE  rcvd: 194

[1] https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/main/controllers/awscluster_controller.go#L291-L298

What did you expect to happen:

To get around this problem, maybe we can wait after the LB is created until the resolution is most likely to succeed (around 2min) before attempting the first DNS resolution. This would decrease the chance of waiting too long for the LB name to resolve due to high TTLs.

Anything else you would like to add:

PR to follow soon.

Environment:

• Cluster-api-provider-aws version: master @ 7ab0780
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale