Refactor Deadline Submission #49

kalisp · 2024-10-18T14:01:28Z

Changelog Description

This PR tries to make DL submissions easier, mostly handling of JobInfo segment. Collection of values is moved out of submission phase much earlier where values could be filled from default values in Settings or overriden by selected list of field by artist in Publisher UI.

Move to universal collector should help with potential validation of filled values.
Profiles in Settings should allow more control of values (like priority, limit_groups) based on task_types or hosts etc.

Additional info

Very WIP, currently working only in AE.
Should be used with ynput/ayon-core#958

Testing notes:

start with this step
follow this step

These settings should be base of generic JobInfo values for Deadline submission. They should contain variables previously contained in Submit* Settings. Some of them should be exposed to Publisher UI as artist overrides.

It was decided that dataclasses should be used instead of attrs. This moves DeadlineJobInfo which is full mapping of JobInfo from abstract submitters to collectors to limit need of new class and necessary remapping later.

Added new argument for old get_job_info (which should be probabaly renamed) to pass base of prepared object to be enhanced with DCC specific fields

Empty strings overrides None defaults which might cause issue (it definitely does for job_delay).

'deadline' dictionary wasnt used at all, it contained large DeadlineJobInfo which just enlarged metadata json unnecessary.

It is handling EnvironmentKey* type of fields

BigRoy

I like where this is going! :)

client/ayon_deadline/lib.py

client/ayon_deadline/abstract_submit_deadline.py

BigRoy · 2024-10-18T15:44:58Z

server/settings/publish_plugins.py

+    chunk_size: int = SettingsField(999, title="Frames per Task")
+    priority: int = SettingsField(50, title="Priority")
+    group: str = SettingsField("", title="Group")
+    limit_groups: list[LimitGroupsSubmodel] = SettingsField(
+        default_factory=list,
+        title="Limit Groups",
+    )
+    concurrent_tasks: int = SettingsField(
+        1, title="Number of concurrent tasks")
+    department: str = SettingsField("", title="Department")
+    use_gpu: bool = SettingsField("", title="Use GPU")
+    job_delay: str = SettingsField(
+        "", title="Delay job",
+        placeholder="dd:hh:mm:ss"
+    )
+    use_published: bool = SettingsField(True, title="Use Published scene")
+    asset_dependencies: bool = SettingsField(True, title="Use Asset dependencies")
+    workfile_dependency: bool = SettingsField(True, title="Workfile Dependency")
+    multiprocess: bool = SettingsField(False, title="Multiprocess")


Would be very nice to get description argument set for each of these.

BigRoy · 2024-10-18T15:50:23Z

server/settings/publish_plugins.py

    enabled: bool = SettingsField(True)
    optional: bool = SettingsField(title="Optional")
    active: bool = SettingsField(title="Active")


With all the other attributes removed from these models we can now make one EnableStateModel class instead that we use for all of these making this file much smaller.

BigRoy · 2024-10-18T15:52:56Z

client/ayon_deadline/plugins/publish/aftereffects/submit_aftereffects_deadline.py

-        dln_job_info.add_render_job_env_var()
+
+        # already collected explicit values for rendered Frames
+        if not dln_job_info.Frames:


I actually think this is a place where we should not be setting this in a Deadline specific way but have it accessible on the instance even outside of the job info. So Frames would be build up from instance.data["frames"] instead so that it could turn out to be non-Deadline specific. However, not critical for now and we can do that later. Might be worth a TODO in code or an issue after this PR.

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

client/ayon_deadline/plugins/publish/global/submit_publish_job.py

server/settings/publish_plugins.py

Not yet working on AOVs.

Deadline portion must be there for ValidateExpectedFiles

client/ayon_deadline/lib.py

iLLiCiTiT · 2024-10-22T08:58:09Z

client/ayon_deadline/lib.py

+
+    def to_json(self) -> str:
+        """Serialize the dataclass instance to a JSON string."""
+        return json.dumps(asdict(self))


Suggested change

return json.dumps(asdict(self))

return json.dumps(asdict(self))

iLLiCiTiT · 2024-10-22T09:14:55Z

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

+
+        # will be reworked when CreateContext contains settings and task types
+        project_name = create_context.project_name
+        project_settings = get_project_settings(project_name)


Suggested change

project_settings = get_project_settings(project_name)

project_settings = (

create_context.get_current_project_settings()

)

iLLiCiTiT · 2024-10-22T09:15:38Z

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

+        if not instance["active"]:  # TODO origin_data seem not right
+            return []
+


I don't think it is necessary to hide attributes on not enabled instance?

Suggested change

if not instance["active"]: # TODO origin_data seem not right

return []

iLLiCiTiT · 2024-10-22T09:18:27Z

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

+        task_name = instance["task"]
+        folder_path = instance["folderPath"]
+        folder_entity = ayon_api.get_folder_by_path(project_name,folder_path)
+        task_entity = ayon_api.get_task_by_name(
+            project_name, folder_entity["id"], task_name)


Note: This happens only to get task type. Fetching this, for each instance, is very slow. We should have option to get task type of an instance from create context -> @kalisp could you create issue?

Suggested change

task_name = instance["task"]

folder_path = instance["folderPath"]

folder_entity = ayon_api.get_folder_by_path(project_name,folder_path)

task_entity = ayon_api.get_task_by_name(

project_name, folder_entity["id"], task_name)

task_name = instance["task"]

folder_path = instance["folderPath"]

folder_entity = ayon_api.get_folder_by_path(project_name,folder_path)

task_entity = ayon_api.get_task_by_name(

project_name, folder_entity["id"], task_name)

iLLiCiTiT · 2024-10-22T09:18:49Z

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

+        defs = []
+
+        defs.extend([
+            UISeparatorDef("deadline_defs_starts"),
+        ])


Suggested change

defs = []

defs.extend([

UISeparatorDef("deadline_defs_starts"),

])

defs = [

UISeparatorDef("deadline_defs_starts"),

]

iLLiCiTiT · 2024-10-22T09:21:49Z

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

+        override_defs = OrderedDict({
+            "chunkSize": NumberDef(
+                "chunkSize",
+                label="Frames Per Task",
+                default=1,
+                decimals=0,
+                minimum=1,
+                maximum=1000
+            ),
+            "priority": NumberDef(
+                "priority",
+                label="Priority",
+                decimals=0
+            ),
+            "department": TextDef(
+                "department",
+                label="Department",
+                default="",
+            ),
+            "limit_groups": TextDef(
+                "limit_groups",
+                label="Limit Groups",
+                default="",
+                placeholder="machine1,machine2"
+            ),
+            "job_delay": TextDef(
+                "job_delay",
+                label="Delay job (timecode dd:hh:mm:ss)",
+                default=""
+            ),
+        })
+        defs = []
+        # The Arguments that can be modified by the Publisher
+        for key, value in override_defs.items():
+            if key not in overrides:
+                continue
+
+            default_value = profile[key]
+            value.default = default_value
+            defs.append(value)
+
+        return defs


Suggested change

override_defs = OrderedDict({

"chunkSize": NumberDef(

"chunkSize",

label="Frames Per Task",

default=1,

decimals=0,

minimum=1,

maximum=1000

),

"priority": NumberDef(

"priority",

label="Priority",

decimals=0

),

"department": TextDef(

"department",

label="Department",

default="",

),

"limit_groups": TextDef(

"limit_groups",

label="Limit Groups",

default="",

placeholder="machine1,machine2"

),

"job_delay": TextDef(

"job_delay",

label="Delay job (timecode dd:hh:mm:ss)",

default=""

),

})

defs = []

# The Arguments that can be modified by the Publisher

for key, value in override_defs.items():

if key not in overrides:

continue

default_value = profile[key]

value.default = default_value

defs.append(value)

return defs

override_defs = [

NumberDef(

"chunkSize",

label="Frames Per Task",

default=1,

decimals=0,

minimum=1,

maximum=1000

),

NumberDef(

"priority",

label="Priority",

decimals=0

),

TextDef(

"department",

label="Department",

default="",

),

TextDef(

"limit_groups",

label="Limit Groups",

default="",

placeholder="machine1,machine2"

),

TextDef(

"job_delay",

label="Delay job (timecode dd:hh:mm:ss)",

default=""

),

]

defs = []

# The Arguments that can be modified by the Publisher

for attr_def in override_defs:

if attr_def.key not in overrides:

continue

attr_def.default = profile[attr_def.key]

defs.append(attr_def)

return defs

iLLiCiTiT · 2024-10-22T09:22:33Z

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

+    @classmethod
+    def on_value_change(cls, event):
+        for instance_change in event["changes"]:
+            if not cls.instance_matches_plugin_families(instance):


instance is not defined variable.

iLLiCiTiT · 2024-10-22T09:23:08Z

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

+            task_name = task_entity["name"]
+            task_type = task_entity["taskType"]
+        profiles = (
+            project_settings["deadline"]


Suggested change

project_settings["deadline"]

project_settings

["deadline"]

iLLiCiTiT · 2024-10-22T09:23:17Z

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

+        project_settings = context_data["project_settings"]
+        task_entity = context_data["taskEntity"]
+
+        task_name = task_type = ""


Suggested change

task_name = task_type = ""

task_name = task_type = None

iLLiCiTiT · 2024-10-22T09:24:51Z

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

+        if profiles:
+            profile = filter_profiles(
+                profiles,
+                {
+                    "host_names": host_name,
+                    "task_types": task_type,
+                    "task_names": task_name,
+                    # "product_type": product_type
+                }
+            )
+            if profile:
+                attr_values = profile


Suggested change

if profiles:

profile = filter_profiles(

profiles,

{

"host_names": host_name,

"task_types": task_type,

"task_names": task_name,

# "product_type": product_type

}

)

if profile:

attr_values = profile

profile = filter_profiles(

profiles,

{

"host_names": host_name,

"task_types": task_type,

"task_names": task_name,

# "product_type": product_type

}

)

if profile:

attr_values = profile

iLLiCiTiT · 2024-10-22T09:25:59Z

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

+    def get_attr_defs_for_instance(cls, create_context, instance):
+        defs = super().get_attr_defs_for_instance(create_context, instance)
+
+        defs.extend([


This will show definitions for all instances in maya.

kalisp added 13 commits October 16, 2024 18:50

New Settings for CollectJobInfoItem

d1a883b

These settings should be base of generic JobInfo values for Deadline submission. They should contain variables previously contained in Submit* Settings. Some of them should be exposed to Publisher UI as artist overrides.

Matched implemented artist overrides definitions

f72e87f

Added collected jobInfo object

2b29403

Use DeadlineJobInfo as collector object

cb0d24d

It was decided that dataclasses should be used instead of attrs. This moves DeadlineJobInfo which is full mapping of JobInfo from abstract submitters to collectors to limit need of new class and necessary remapping later.

Removed six dependency

37c0a0d

Implemented new generic method applicable for all DCC

3358838

Added new argument for old get_job_info (which should be probabaly renamed) to pass base of prepared object to be enhanced with DCC specific fields

Removed unneeded imports

60da8b9

Do not pass empty string to DeadlineJobInfo

ae3c739

Empty strings overrides None defaults which might cause issue (it definitely does for job_delay).

Removed unneded imports

1c8990a

Reimplemented get_job_info, moved to generic impl

57860ce

Removed storign deadline info to instance

10f566f

'deadline' dictionary wasnt used at all, it contained large DeadlineJobInfo which just enlarged metadata json unnecessary.

Updates to DeadlineJobInfo to match previous implementation

bbb12b7

It is handling EnvironmentKey* type of fields

Added Frames field for explicit rendering

c26c997

kalisp requested review from BigRoy, antirotor and iLLiCiTiT October 18, 2024 14:01

kalisp self-assigned this Oct 18, 2024

BigRoy reviewed Oct 18, 2024

View reviewed changes

kalisp added 8 commits October 18, 2024 18:15

Run CollectJobInfo only locally

e1eba7f

Implemented explicit frames filtering on simple files

8062462

Not yet working on AOVs.

Reworked removal of deadline in submit job

ec908e0

Deadline portion must be there for ValidateExpectedFiles

Removed empty line

64b22ff

Added empty line at the end

0d3125d

Formatting

c206b24

Removed forgotten dev code

803c25c

Refactor batchname

201c5b8

BigRoy added the type: enhancement Improvement of existing functionality or minor addition label Oct 21, 2024

iLLiCiTiT reviewed Oct 22, 2024

View reviewed changes

client/ayon_deadline/lib.py Show resolved Hide resolved

iLLiCiTiT reviewed Oct 22, 2024

View reviewed changes

BigRoy mentioned this pull request Oct 23, 2024

Render product publish: use same Deadline options as other DCCs ynput/ayon-blender#63

Open

2 tasks

BigRoy linked an issue Oct 23, 2024 that may be closed by this pull request

Render product publish: use same Deadline options as other DCCs ynput/ayon-blender#63

Open

2 tasks

iLLiCiTiT mentioned this pull request Oct 23, 2024

Create Context: Helper function to get entities ynput/ayon-core#967

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor Deadline Submission #49

Refactor Deadline Submission #49

kalisp commented Oct 18, 2024 •

edited

Loading

BigRoy left a comment

BigRoy Oct 18, 2024

BigRoy Oct 18, 2024

BigRoy Oct 18, 2024

iLLiCiTiT Oct 22, 2024

iLLiCiTiT Oct 22, 2024

iLLiCiTiT Oct 22, 2024

iLLiCiTiT Oct 22, 2024

iLLiCiTiT Oct 22, 2024

iLLiCiTiT Oct 22, 2024

iLLiCiTiT Oct 22, 2024

iLLiCiTiT Oct 22, 2024

iLLiCiTiT Oct 22, 2024

iLLiCiTiT Oct 22, 2024

iLLiCiTiT Oct 22, 2024

	return json.dumps(asdict(self))
	return json.dumps(asdict(self))

-        project_settings = get_project_settings(project_name)
+        project_settings = (
+            create_context.get_current_project_settings()
+        )

		if not instance["active"]: # TODO origin_data seem not right
		return []

Refactor Deadline Submission #49

Are you sure you want to change the base?

Refactor Deadline Submission #49

Conversation

kalisp commented Oct 18, 2024 • edited Loading

Changelog Description

Additional info

Testing notes:

BigRoy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kalisp commented Oct 18, 2024 •

edited

Loading