Topics validation can be done via Kafka Admin Client. #79
Conversation
| .refreshAfterWrite(Duration.ofSeconds(30)) | ||
| .build(this::fetchTopicDescription); | ||
|
|
||
| CaffeineCacheMetrics.monitor(meterRegistry, topicDescriptionsCache, "tkmsTopicDescriptions"); |
There was a problem hiding this comment.
According to the Javadoc we need to call .recordStats() in the Caffeine builder above in order to gather non-zero stats.
|
|
||
| ### Changed | ||
|
|
||
| - Topic validation can now be done via Kafka Admin, instead of Kafka Producer. |
There was a problem hiding this comment.
What's the upside of doing these topic validations before publishing messages instead of just failing with failed publishing?
There was a problem hiding this comment.
When publishing a message on shard-partition fails, then that shard-partition stops.
I.e. we have a poison-pill situtation.
The reason for doing different validation on message registering is to lower the probability of that situation happening.
| tkmsMetricsTemplate.registerNoAclOperationsFetched(shardPartition, topic); | ||
| } else if (!aclOperations.contains(AclOperation.ALL) | ||
| && !aclOperations.contains(AclOperation.WRITE) | ||
| && !aclOperations.contains(AclOperation.IDEMPOTENT_WRITE) |
There was a problem hiding this comment.
IDEMPOTENT_WRITE is not enough by itself, we need WRITE or ALL too. That is IDEMPOTENT_WRITE does not include WRITE implicitly. We have a catch all rule for all clusters to allow everyone to do I_W: https://github.com/transferwise/kafka-topic-configurations/blob/2392c8d3289af69254bcd5cecd0c8e8770b70428/main/production/acls/cluster/cluster-acls.yml#L25-L27
| /** | ||
| * Uses AdminClient to validate topics. | ||
| * | ||
| * <p>AdminClient would allow us to also check if topics have suitable ACLs. |
There was a problem hiding this comment.
| * <p>AdminClient would allow us to also check if topics have suitable ACLs. | |
| * <p>AdminClient allows us to also check if topics have suitable ACLs. |
Context
Changed
The new logic is under a feature flag, until the logic gets more battle tested.
Checklist