Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Logs are missing during high loads #945

Closed
beth2016 opened this issue Oct 4, 2023 · 2 comments
Closed

Logs are missing during high loads #945

beth2016 opened this issue Oct 4, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@beth2016
Copy link

beth2016 commented Oct 4, 2023

What happened?

Description

When we run performance tests (40k requests/min) , we see logs are intermittently dropped for 30-90 seconds. This issue started manifesting after we upgrade otel from 0.43 to 0.81.

Steps to Reproduce

Expected Result

Actual Result

Chart version

0.81

Environment information

Environment

Cloud: Openshift
k8s version: 1.23

Chart configuration

No response

Log output

No response

Additional context

No response
@beth2016 beth2016 added the bug Something isn't working label Oct 4, 2023
@matthewmodestino
Copy link

Hey Beth! Hope all is well!

This could be caused by file rotation under high load on the node, or other scenarios.

Any logs from the agents affected that may help?

@atoulme
Copy link
Contributor

atoulme commented Nov 1, 2023

You might be best served by opening a support case to follow up. Please post the support case ID in this issue so we can work with you directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@atoulme @beth2016 @matthewmodestino