Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

improve prometheus gpu_type fetching #90

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

improve prometheus gpu_type fetching #90

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d528d82
improve prometheus gpu_type fetching
nurbal Nov 28, 2023
865b2f7
Merge branch 'master' into fix_prometheus_gpu_type
nurbal Nov 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion sarc/jobs/sacct.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,7 +286,6 @@ def update_allocated_gpu_type(cluster: ClusterConfig, entry: SlurmJob) -> Option
output = get_job_time_series(
job=entry,
metric="slurm_job_utilization_gpu_memory",
max_points=1,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

J'ai peur que de carrément enlever max_points rende le query beaucoup plus couteux pour les longues jobs. Est-ce que d'augmenter max_points pourrait régler le problème? C'est un peu bizarre aussi comme problème, peut-être qu'il y a un bug dans get_job_time_series avec l'argument max_points.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Essais sur la journée du 3 septembre sur le cluster Mila:
sans limite de points:

From 138 jobs with prometheus stats:
97 jobs with gpu_type detected

max_points=1 :

From 138 jobs with prometheus stats:
36 jobs with gpu_type detected

max_points=2 :

From 138 jobs with prometheus stats:
58 jobs with gpu_type detected

Donc non, je ne crois pas qu'augmenter max_points à une valeur arbitraire règle le problème.

Je vais regarder si get_job_time_series n'est pas bugguée...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Le code source de get_job_time_series a comme paramètres par défaut:

    min_interval: int = 30,
    max_points: int = 100,

Ce qui signifie que tout job inférieur à 30s a des chances de passer entre les mailles du filet.

Je vais voir si je peux modifier l’appel à prometheus pour les cas où max_points=1 ou max_points=0 afin d'être certain de récupérer des informations sur le job, et aussi tester si avec des valeurs plus faibles pour min_interval on obtient d'autres résultats...

dataframe=False,
)
if output:
Expand Down