Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ECS Fargate CPU usage calculated incorrectly? #6

Open
marksumm opened this issue Nov 17, 2021 · 2 comments
Open

ECS Fargate CPU usage calculated incorrectly? #6

marksumm opened this issue Nov 17, 2021 · 2 comments

Comments

@marksumm
Copy link

marksumm commented Nov 17, 2021

I did some experiments running instances of containerstack/cpustress in ECS Fargate on Linux with a task CPU limit of 1 vCPU (1024 shares). The instances were configured to always use 100% of the available CPU resources, by running with arguments --cpu=2 to start 2 worker threads and --timeout=86400 to prevent early termination. I've noticed some strange behaviour from the exporter...

  1. ecs_task_cpu_limit is returned correctly in CPU shares (1024)

  2. ecs_task_cpu_usage_ratio is correctly scaled between 0 and 1 when exported_container_name="task"

  3. ecs_task_cpu_usage_ratio is returned in CPU shares and not scaled between 0 and 1 when exported_container_name != "task". Also, it seems to be out by a factor of 2.

@marksumm
Copy link
Author

marksumm commented Nov 17, 2021

I think I've found the cause of the issues: The container CPU limit is being returned as 2 (presumably due to default Linux cgroup behaviour) when it's actually 0 in the ECS task definition. Causes normalize_cpu_usage to apply case 4 scaling, when case 3 is expected.

@marksumm
Copy link
Author

After experimenting with different CPU limits on individual containers, I've realised that the values returned for ecs_task_percpu_usage_ratio are inexplicable. I would expect them to be affected by the same issues as ecs_task_cpu_usage_ratio and also to sum to the same value, but it appears not to be the case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant