attempt to commonize Kubeflow jobs Multikueue support methods #2795

mszadkow · 2024-08-07T15:27:10Z

/kind cleanup

What type of PR is this?

What this PR does / why we need it:

Attempt to unify code so it's not repeated for each kubeflow job multikueue adapter

Which issue(s) this PR fixes:

Relates #2552

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

k8s-ci-robot · 2024-08-07T15:27:12Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

netlify · 2024-08-07T15:27:26Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`5ed0f3b`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/66c36a3de9eb690008605005

mszadkow · 2024-08-07T15:31:34Z

/ok-to-test

pkg/controller/jobframework/interface.go

pkg/controller/jobs/kubeflow/common/common.go

pkg/controller/jobframework/interface.go

pkg/controller/jobs/kubeflow/common/common.go

tenzen-y

That doesn't seem to be my expectations.
Could we commonize kubeflow adapters like kubeflow job reconcilers?

https://github.com/kubernetes-sigs/kueue/tree/main/pkg/controller/jobs/kubeflow/kubeflowjob

pkg/controller/jobframework/interface.go

test/integration/multikueue/multikueue_test.go

pkg/controller/jobs/kubeflow/common/common.go

mszadkow · 2024-08-16T11:41:57Z

@alculquicondor you may want to have a look here too ;)

alculquicondor

looks good overall :)

pkg/controller/jobs/kubeflow/kubeflowjob/kubeflowjob_multikueue_adapter.go

alculquicondor · 2024-08-16T18:02:51Z

pkg/controller/jobs/kubeflow/kubeflowjob/kubeflowjob_multikueue_adapter.go

+	// add the prebuilt workload
+	labels := remoteJob.GetLabels()
+	if remoteJob.GetLabels() == nil {
+		labels = make(map[string]string, 2)
+	}
+	labels[constants.PrebuiltWorkloadLabel] = workloadName
+	labels[kueuealpha.MultiKueueOriginLabel] = origin
+	remoteJob.SetLabels(labels)
+


I think this code should even be part of the MK workload controller, but that can be left for a follow up.

alculquicondor · 2024-08-16T18:03:56Z

/release-note-edit

NONE

tenzen-y

Overall, looks good to me.
This is a comment for a followup.

Could you work on consolidating / commonizing the kubeflow MK integration tests (TFJob/PaddleJob/XGBoostJob/PyTorchJob)

kueue/test/integration/multikueue/multikueue_test.go

Line 781 in 5b9ff8e

ginkgo.It("Should run a TFJob on worker if admitted", func() {

?

I'm wondering if we can reduce the kubeflow MK integration tests like kubeflowjob reconcilers.

pkg/controller/jobs/kubeflow/kubeflowjob/kubeflowjob_multikueue_adapter.go

mszadkow · 2024-08-19T13:32:45Z

Overall, looks good to me. This is a comment for a followup.

Could you work on consolidating / commonizing the kubeflow MK integration tests (TFJob/PaddleJob/XGBoostJob/PyTorchJob)

kueue/test/integration/multikueue/multikueue_test.go

Line 781 in 5b9ff8e

ginkgo.It("Should run a TFJob on worker if admitted", func() {

?
I'm wondering if we can reduce the kubeflow MK integration tests like kubeflowjob reconcilers.

I actually work on this one while doing MPIJob multikueue adapter, but I can add it here.

tenzen-y · 2024-08-19T13:40:13Z

Overall, looks good to me. This is a comment for a followup.
Could you work on consolidating / commonizing the kubeflow MK integration tests (TFJob/PaddleJob/XGBoostJob/PyTorchJob)

kueue/test/integration/multikueue/multikueue_test.go

Line 781 in 5b9ff8e

ginkgo.It("Should run a TFJob on worker if admitted", func() {

?
I'm wondering if we can reduce the kubeflow MK integration tests like kubeflowjob reconcilers.

I actually work on this one while doing MPIJob multikueue adapter, but I can add it here.

It seems that this PR will be finalized soon. So, it may be helpful to work on test refactoring at another PR so that we do not block merging this PR.

alculquicondor

/lgtm
/approve

k8s-ci-robot · 2024-08-19T16:17:14Z

LGTM label has been added.

Git tree hash: 282c50d2fe8cd61ed799f6f3196028fd327700e9

k8s-ci-robot · 2024-08-19T16:17:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, mszadkow

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/controller/OWNERS~~ [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…etes-sigs#2795) * attempt to commonize Kubeflow jobs Multikueue support methods * make generic adapter to be specialized by job type * Move common Multikueue adapter to Kubeflowjob package * Changes after code review

attempt to commonize Kubeflow jobs Multikueue support methods

e63a990

k8s-ci-robot requested review from denkensk and trasc August 7, 2024 15:27

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 7, 2024

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Aug 7, 2024

trasc reviewed Aug 8, 2024

View reviewed changes

tenzen-y reviewed Aug 9, 2024

View reviewed changes

pkg/controller/jobframework/interface.go Outdated Show resolved Hide resolved

make generic adapter to be specialized by job type

f065e3f

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 13, 2024

trasc reviewed Aug 13, 2024

View reviewed changes

test/integration/multikueue/multikueue_test.go Outdated Show resolved Hide resolved

pkg/controller/jobs/kubeflow/common/common.go Outdated Show resolved Hide resolved

Move common Multikueue adapter to Kubeflowjob package

7140be8

mszadkow marked this pull request as ready for review August 14, 2024 12:38

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 14, 2024

k8s-ci-robot requested review from mimowo and trasc August 14, 2024 12:38

alculquicondor reviewed Aug 16, 2024

View reviewed changes

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Aug 16, 2024

tenzen-y reviewed Aug 16, 2024

View reviewed changes

pkg/controller/jobs/kubeflow/kubeflowjob/kubeflowjob_multikueue_adapter.go Outdated Show resolved Hide resolved

Changes after code review

5ed0f3b

mszadkow force-pushed the feature/support-kubeflow-multikueue-common-interface branch from 5f98f05 to 5ed0f3b Compare August 19, 2024 15:52

alculquicondor reviewed Aug 19, 2024

View reviewed changes

k8s-ci-robot assigned alculquicondor Aug 19, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 19, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 19, 2024

k8s-ci-robot merged commit 801734b into kubernetes-sigs:main Aug 19, 2024
16 checks passed

k8s-ci-robot added this to the v0.9 milestone Aug 19, 2024

mbobrovskyi deleted the feature/support-kubeflow-multikueue-common-interface branch August 19, 2024 17:06

tenzen-y mentioned this pull request Aug 20, 2024

Skip flaky e2e MultiKueue test for XGBoost #2861

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

attempt to commonize Kubeflow jobs Multikueue support methods #2795

attempt to commonize Kubeflow jobs Multikueue support methods #2795

mszadkow commented Aug 7, 2024 •

edited by k8s-ci-robot

Loading

k8s-ci-robot commented Aug 7, 2024

netlify bot commented Aug 7, 2024 •

edited

Loading

mszadkow commented Aug 7, 2024

tenzen-y left a comment

mszadkow commented Aug 16, 2024

alculquicondor left a comment

alculquicondor Aug 16, 2024

mszadkow Aug 19, 2024

alculquicondor commented Aug 16, 2024

tenzen-y left a comment

mszadkow commented Aug 19, 2024

tenzen-y commented Aug 19, 2024

alculquicondor left a comment

k8s-ci-robot commented Aug 19, 2024

k8s-ci-robot commented Aug 19, 2024

attempt to commonize Kubeflow jobs Multikueue support methods #2795

attempt to commonize Kubeflow jobs Multikueue support methods #2795

Conversation

mszadkow commented Aug 7, 2024 • edited by k8s-ci-robot Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Aug 7, 2024

netlify bot commented Aug 7, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

mszadkow commented Aug 7, 2024

tenzen-y left a comment

Choose a reason for hiding this comment

mszadkow commented Aug 16, 2024

alculquicondor left a comment

Choose a reason for hiding this comment

alculquicondor Aug 16, 2024

Choose a reason for hiding this comment

mszadkow Aug 19, 2024

Choose a reason for hiding this comment

alculquicondor commented Aug 16, 2024

tenzen-y left a comment

Choose a reason for hiding this comment

mszadkow commented Aug 19, 2024

tenzen-y commented Aug 19, 2024

alculquicondor left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 19, 2024

k8s-ci-robot commented Aug 19, 2024

mszadkow commented Aug 7, 2024 •

edited by k8s-ci-robot

Loading

netlify bot commented Aug 7, 2024 •

edited

Loading