Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CPU/RAM overload due to query with regexp and metadata #14403

Open
ddday999 opened this issue Oct 7, 2024 · 0 comments
Open

CPU/RAM overload due to query with regexp and metadata #14403

ddday999 opened this issue Oct 7, 2024 · 0 comments

Comments

@ddday999
Copy link

ddday999 commented Oct 7, 2024

Hello, dears!
Please, help me to understand the strange behaviour of Loki.

Describe the bug
Small query with construction <regexp + metadata + regexp> in Grafana Explore overloads RAM and CPU await
{application="app"} !~ "(?i)info" | filename = "E:/app/app.log" |~ "(?i)err"

RAM:
Image

CPU:
Image

But, occured that constructions below:

  1. {application="app"} !~ "(?i)info"
  2. {application="app"} !~ "(?i)info" |~ "(?i)err"
  3. {application="app"} !~ "(?i)info" |~ "(?i)err" | filename = "E:/app/app.log"
  4. {application="app"} | filename = "E:/app/app.log" !~ "(?i)info" |~ "(?i)err"

works properly - without any overloads by RAM/CPU

Only construction <regexp + metadata + regexp> triggers this problem.

In addition, in scanario below didnt stops the overload:
Cancellation of query (by timeout or manually) in Grafana, then OOM kill of Loki container - its restarting with and overloading again

Helps only manual restart of Loki container

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki
  2. Started Grafana
  3. Started S3 - MinIO
  4. Query: {application="app"} !~ "(?i)info" | filename = "E:/app/app.log" |~ "(?i)err"

Expected behavior
Overload by RAM/CPU await

Environment:

  1. Monolithic Loki deployment (in cluster of 3 nodes: 8 CPU/12G RAM, 4G Swap)
  2. Running in container (every Loki container limited by 6 CPU/10G RAM, 4G Swap)
  3. Logs stores in MinIO S3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant