Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Kafka *ByteRate metrics are non-zero for idle pipelines #3180

Open
cmanning09 opened this issue Aug 16, 2023 · 1 comment
Open

[BUG] Kafka *ByteRate metrics are non-zero for idle pipelines #3180

cmanning09 opened this issue Aug 16, 2023 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@cmanning09
Copy link
Contributor

Describe the bug
While my pipeline is idle the kafka source plugin continuously reports byte rate data when there is no data flowing from the kafka cluster to my pipeline. This is seen for both incomingByteRate and outgoingByteRate.

Ideally I would expect this to be zero during idle periods.

@cmanning09 cmanning09 added bug Something isn't working untriaged labels Aug 16, 2023
@cmanning09 cmanning09 changed the title [BUG] Kafka *ByteRate metric is non zero while pipeline is not ingesting data. [BUG] Kafka *ByteRate metrics are non-zero for idle pipelines Aug 16, 2023
@dlvenable dlvenable added this to the v2.4 milestone Aug 16, 2023
@kkondaka
Copy link
Collaborator

incomingByteRate and outgoingByteRate are obtained from the Kafka Client Library by doing consumer.metrics(). I think they are non-zero because there are house-keeping messages like heart-beat messages are sent between the client and the server even when there is no regular traffic. DataPrepper just exposes whatever value is returned by consumer.metrics()

@dlvenable dlvenable modified the milestones: v2.4, v2.5 Aug 22, 2023
@dlvenable dlvenable modified the milestones: v2.5, v2.6 Oct 6, 2023
@dlvenable dlvenable removed this from the v2.6 milestone Nov 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

No branches or pull requests

3 participants