-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Houdini: Publish any ROP node (Generic Creator) #542
Conversation
…ion on string parms by default
… creator for Houdini
…`CreatedInstance` data
Instances are not unique error in demo fileSo in my recording I hit this issue with the low-level PDG example:
With that demo scene you'll hit the same error because those PDG nodes cook in-process so the logic runs in the current houdini process. Since it triggers a regular So basically even without the dynamic instances the
Deleting those nodes for model and pointcache from the workfile will fix that and allow you to test it further. Note that the workfile will always be collected (but the instance could technically be deactivated through the instance, etc.) Anyway, something to be aware of. (You could technically also open the publisher UI, toggle off all instances, click save and close it again so that they are off by default when the CreateContext finds them) The better solution would be to run the cook for those PDG/TOPs nodes Cook (Out-of-process) but you'll need to make sure that those run in the same outer process so they have access to the exact python objects that is the TODO
|
Hey, Also, few notes/questions:
I find it cool as low level publishing that by passes all validators to just publish the exported data.
2024-05-23.17-29-53.mp4 |
What about converting the product type Houdini parameter in generic instances to a drop down menu. from ayon_core.pipeline import registered_host
from ayon_core.pipeline.create import CreateContext
import hou
host = registered_host()
assert host, "No registered host."
create_context = CreateContext(host)
product_types = set()
for _, creator in create_context.manual_creators.items():
if creator.product_type == "generic":
continue
elif creator.product_type in {
"karma_rop", "mantra_rop", "arnold_rop",
"redshift_rop", "vray_rop", "usdrender"
}:
product_types.add("render")
else:
product_types.add(creator.product_type)
print(product_types) |
Here's my demo(s)! demo_first_part.mp4I had to re-record the end because my microphone did a prank on me too demo_second_part.mp4 |
I'm now realizing that I forgot to touch on what the And I'm not sure if I mentioned this point on my second recording but the main thing here is that this pipeline I showed is production ready, all the tools that artists need to render/cache in the farm and publish all types are there. This is what an MVP (minimum viable product) is, after having this, then you can start adding complexity, but we should never start from a complex system and work all the way down IMO |
Hey, Here's screen shot after seeing Some demo 2024-05-24.01-28-56.mp4 |
Just want to say thanks @fabiaserra @MustafaJafar - those are great details! I like:
I dislike (some from the demos, some are comments on how things currently work in AYON):
So as @MustafaJafar stated:
Yes! But preferably in a way that we don't need to go create complex creators all over the place, etc. Also got some comments from Alexandru Preoteasa (Static VFX):
Which made me wonder @fabiaserra did you end up doing that in your setups as well? I suppose just a regular Deadline submissions, and then publishing.@fabiaserra What's a bit unclear from your video is that you're using the regular deadline submission. Does that automatically also publish the output in a separate deadline job that gets created by the deadline submitter because you "hacked that into it"? Or how does that work? It seems you're basically submitting it to Deadline on its own - basically close to no AYON related logic to that, but then when/how does that publish through AYON? Flipbook toolAlso really like the Flipbook tool you showed in the Part2 video at 03:30 with how easy it is to create the preview, check it and then publish it directly there. |
Thank you @BigRoy for summarizing your highlights and trying to transform these into action items! Let me try respond to some of your points.
I agree that would be nice to close the bridge with making use of the existing publish dialog and plugin system (although quite a lot of Houdini TDs would still prefer to submit their publishes through the node graph like we do with the
I agree and I guess this touches my prior point. We created our own separate API that abstracts the process of creating the .json file to run headless publishes and I keep using on all the tools that I have been building for our needs and I find it extremely useful. This stems from the fact that when I needed to create these tools I found it very hard to dissect the publish framework to run it in a simple manner from other places. If I had understood then how to do it through the system, I wouldn't have created this separate abstraction. I think the batch publishing introduced here kind of goes into this direction but I still find it too verbose and rigid, like mentioned by @BigRoy here https://gist.github.com/MustafaJafar/bd2a388e4a6aa3613d64a186ebb6660c?permalink_comment_id=5067094#gistcomment-5067094
Yeah, unless the validators are 100% reliable and cover all use cases (quite impossible in reality, specially when AYON tries to fit a lot of workflows from different studios), it adds extra blockers to using the system. All validators should be optional and the publish dialog shouldn't show you ALL the plugins that exist in the system because it's very hard to know which plugins are actually required for that publish instance and they add too much noise to make the dialog actually useful to read.
Yeah definitely, although that would be a very easy addition to the abstracted publish API to collect inputs on submission (reusing the same logic of the existing AYON plugin for doing that).
Please!
In my publish abstraction module, for image type sequences (i.e.
Correct, we have two different Deadline submission entry points. I haven't hacked the vanilla Houdini Deadline submission yet to make it understand AYON and automatically create publish tasks as dependency tasks, although it would be certainly possible and not too hard to do. However, we currently are on the philosophy that artists need to always validate the things that get published so we wouldn't want to run the publish tasks on the same render submission. The two entry points are:
The tool is basically this right now, the code is not the cleanest as it was mostly just a copy paste from an old tool we had in our legacy pipeline to have an MVP they can start using already: import os
import logging
from qtpy import QtWidgets, QtGui
import hou
from ayon_core.lib import path_tools
from ayon_core.modules.deadline.lib import publish
log = logging.getLogger(__name__)
class FlipbookDialog(QtWidgets.QDialog):
def __init__(self, parent=None):
QtWidgets.QDialog.__init__(self, parent)
self.scene_viewer = hou.ui.paneTabOfType(hou.paneTabType.SceneViewer)
# other properties
self.setWindowTitle("Flipbook")
# define general layout
layout = QtWidgets.QVBoxLayout()
groupLayout = QtWidgets.QVBoxLayout()
# output toggles
self.outputToMplay = QtWidgets.QCheckBox("MPlay Output", self)
self.outputToMplay.setChecked(True)
self.beautyPassOnly = QtWidgets.QCheckBox("Beauty Pass", self)
self.useMotionblur = QtWidgets.QCheckBox("Motion Blur", self)
# description widget
self.descriptionLabel = QtWidgets.QLabel("Description")
self.description = QtWidgets.QLineEdit()
resolution = self.get_default_resolution()
# resolution sub-widgets x
self.resolutionX = QtWidgets.QWidget()
resolutionXLayout = QtWidgets.QVBoxLayout()
self.resolutionXLabel = QtWidgets.QLabel("Width")
self.resolutionXLine = QtWidgets.QLineEdit(resolution[0])
resolutionXLayout.addWidget(self.resolutionXLabel)
resolutionXLayout.addWidget(self.resolutionXLine)
self.resolutionX.setLayout(resolutionXLayout)
# resolution sub-widgets y
self.resolutionY = QtWidgets.QWidget()
resolutionYLayout = QtWidgets.QVBoxLayout()
self.resolutionYLabel = QtWidgets.QLabel("Height")
self.resolutionYLine = QtWidgets.QLineEdit(resolution[1])
resolutionYLayout.addWidget(self.resolutionYLabel)
resolutionYLayout.addWidget(self.resolutionYLine)
self.resolutionY.setLayout(resolutionYLayout)
output_path = self.get_output_path()
self.outputLabel = QtWidgets.QLabel(
f"Flipbooking to: {output_path}"
)
# resolution group
self.resolutionGroup = QtWidgets.QGroupBox("Resolution")
resolutionGroupLayout = QtWidgets.QHBoxLayout()
resolutionGroupLayout.addWidget(self.resolutionX)
resolutionGroupLayout.addWidget(self.resolutionY)
self.resolutionGroup.setLayout(resolutionGroupLayout)
# frame range widget
self.frameRange = QtWidgets.QGroupBox("Frame range")
frameRangeGroupLayout = QtWidgets.QHBoxLayout()
# frame range start sub-widget
self.frameRangeStart = QtWidgets.QWidget()
frameRangeStartLayout = QtWidgets.QVBoxLayout()
self.frameRangeStartLabel = QtWidgets.QLabel("Start")
self.frameRangeStartLine = QtWidgets.QLineEdit("$RFSTART")
frameRangeStartLayout.addWidget(self.frameRangeStartLabel)
frameRangeStartLayout.addWidget(self.frameRangeStartLine)
self.frameRangeStart.setLayout(frameRangeStartLayout)
frameRangeGroupLayout.addWidget(self.frameRangeStart)
# frame range end sub-widget
self.frameRangeEnd = QtWidgets.QWidget()
frameRangeEndLayout = QtWidgets.QVBoxLayout()
self.frameRangeEndLabel = QtWidgets.QLabel("End")
self.frameRangeEndLine = QtWidgets.QLineEdit("$RFEND")
frameRangeEndLayout.addWidget(self.frameRangeEndLabel)
frameRangeEndLayout.addWidget(self.frameRangeEndLine)
self.frameRangeEnd.setLayout(frameRangeEndLayout)
frameRangeGroupLayout.addWidget(self.frameRangeEnd)
self.frameRange.setLayout(frameRangeGroupLayout)
# copy to path widget
self.copyPathButton = QtWidgets.QPushButton("Copy Path to Clipboard")
# options group
self.optionsGroup = QtWidgets.QGroupBox("Flipbook options")
groupLayout.addWidget(self.outputToMplay)
groupLayout.addWidget(self.beautyPassOnly)
groupLayout.addWidget(self.useMotionblur)
groupLayout.addWidget(self.copyPathButton)
self.optionsGroup.setLayout(groupLayout)
# button box buttons
self.cancelButton = QtWidgets.QPushButton("Cancel")
self.startButton = QtWidgets.QPushButton("Start Flipbook")
self.publishButton = QtWidgets.QPushButton("Submit to Publish")
self.publishButton.setEnabled(
os.path.exists(hou.expandString(output_path))
)
# lower right button box
buttonBox = QtWidgets.QDialogButtonBox()
buttonBox.addButton(self.startButton, QtWidgets.QDialogButtonBox.ActionRole)
buttonBox.addButton(self.cancelButton, QtWidgets.QDialogButtonBox.ActionRole)
buttonBox.addButton(self.publishButton, QtWidgets.QDialogButtonBox.ActionRole)
# widgets additions
layout.addWidget(self.outputLabel)
layout.addWidget(self.descriptionLabel)
layout.addWidget(self.description)
layout.addWidget(self.frameRange)
layout.addWidget(self.resolutionGroup)
layout.addWidget(self.optionsGroup)
layout.addWidget(buttonBox)
# connect button functionality
self.cancelButton.clicked.connect(self.close_window)
self.startButton.clicked.connect(self.start_flipbook)
self.publishButton.clicked.connect(self.submit_to_publish)
self.copyPathButton.clicked.connect(self.copy_path_to_clipboard)
self.description.textChanged.connect(self.update_output_path)
# finally, set layout
self.setLayout(layout)
def close_window(self):
self.close()
def update_output_path(self):
output_path = self.get_output_path()
self.outputLabel.setText(f"Flipbooking to: {output_path}")
self.publishButton.setEnabled(
os.path.exists(hou.expandString(output_path))
)
# get a flipbook settings object and return with given inputs
def get_flipbook_settings(self, input_settings):
settings = self.scene_viewer.flipbookSettings().stash()
log.info("Using '%s' object", settings)
# standard settings
settings.outputToMPlay(input_settings["mplay"])
settings.output(input_settings["output"])
settings.useResolution(True)
settings.resolution(input_settings["resolution"])
settings.cropOutMaskOverlay(True)
settings.frameRange(input_settings["frameRange"])
settings.beautyPassOnly(input_settings["beautyPass"])
settings.antialias(hou.flipbookAntialias.HighQuality)
settings.sessionLabel(input_settings["sessionLabel"])
settings.useMotionBlur(input_settings["motionBlur"])
return settings
def get_output_path(self, expand=False):
description = self.description.text().replace(" ", "_")
path = "$HIP/flipbook/$HIPNAME/flipbook{}.$F4.jpg".format(
f"_{description}" if description else ""
)
if expand:
path = hou.expandString(path)
return path
def get_default_resolution(self):
cam = self.scene_viewer.curViewport().camera()
if not cam: # Use the main render_cam if no viewport cam
cam = hou.node("/obj/render_cam")
if cam:
x = cam.parm("resx").eval()
y = float(cam.parm("resy").eval())
pixel_ratio = cam.parm("aspect").eval()
return (x, int(y / pixel_ratio))
return ("$RESX", "$RESY")
def start_flipbook(self):
inputSettings = {}
# validation of inputs
inputSettings["frameRange"] = self.get_frame_range()
inputSettings["resolution"] = self.get_resolution()
inputSettings["mplay"] = self.outputToMplay.isChecked()
inputSettings["beautyPass"] = self.beautyPassOnly.isChecked()
inputSettings["motionBlur"] = self.useMotionblur.isChecked()
outputPath = self.get_output_path()
inputSettings["output"] = outputPath
inputSettings["sessionLabel"] = outputPath
log.info("Using the following settings, %s", inputSettings)
base_dir = os.path.dirname(hou.expandString(outputPath))
if not os.path.exists(base_dir):
os.makedirs(base_dir)
# retrieve full settings object
settings = self.get_flipbook_settings(inputSettings)
# run the actual flipbook
try:
with hou.InterruptableOperation(
"Flipbooking",
long_operation_name="Creating a flipbook",
open_interrupt_dialog=True,
) as operation:
operation.updateLongProgress(0.25, "Starting Flipbook")
hou.SceneViewer.flipbook(self.scene_viewer, settings=settings)
operation.updateLongProgress(1, "Flipbook successful")
# self.close_window()
except Exception as e:
log.error("Oops, something went wrong!")
log.error(e)
return
self.publishButton.setEnabled(
os.path.exists(hou.expandString(outputPath))
)
def submit_to_publish(self):
output_path = self.get_output_path(expand=True)
product_name = os.path.basename(output_path).split(".")[0]
# Add task name suffix to product name
product_name = f"{product_name}_{os.getenv('AYON_TASK_NAME')}"
if not os.path.exists(output_path):
hou.ui.displayMessage(
f"Flipbook path {output_path} does not exist, generate it first.",
title="Path does not exist",
severity=hou.severityType.Error,
)
button_idx, values = hou.ui.readMultiInput(
"Publish Input",
input_labels=("Comment", "Version (optional)"),
buttons=("Submit", "Cancel"),
default_choice=0,
close_choice=1,
initial_contents=(
"",
path_tools.get_version_from_path(hou.hipFile.basename())
)
)
if button_idx:
return
comment, version = values
publish_data = {"out_colorspace": "rec709"}
if comment:
publish_data["comment"] = comment
if version:
publish_data["version"] = int(version)
message, success = publish.publish_version(
os.getenv("AYON_PROJECT_NAME"),
os.getenv("AYON_FOLDER_PATH"),
os.getenv("AYON_TASK_NAME"),
"review",
product_name,
{"jpg": output_path},
publish_data=publish_data,
overwrite_version=True if values[1] else False,
)
if success:
hou.ui.displayMessage(
message,
title="Submission successful",
severity=hou.severityType.Message,
)
else:
hou.ui.displayMessage(
message,
title="Submission error",
severity=hou.severityType.Error,
)
# copyPathButton callback
# copy the output path to the clipboard
def copy_path_to_clipboard(self):
path = self.get_output_path(expand=True)
log.info("Copying path to clipboard: %s", path)
QtGui.QGuiApplication.clipboard().setText(path)
def get_frame_range(self):
return (
int(hou.expandString(self.frameRangeStartLine.text())),
int(hou.expandString(self.frameRangeEndLine.text())),
)
def get_resolution(self):
return (
int(hou.expandString(self.resolutionXLine.text())),
int(hou.expandString(self.resolutionYLine.text())),
)
def show_dialog(parent):
dialog = FlipbookDialog(parent)
dialog.show() |
…skip_rendering_if_local Houdini: add 'generic' family to collect farm instances and skip render if local publishing.
…expose_generic_creator Houdini: Add 'Make AYON Publishable' to OPMenu
…ancement/houdini_generic_publish # Conflicts: # server_addon/houdini/client/ayon_houdini/plugins/create/create_dynamic.py # server_addon/houdini/client/ayon_houdini/plugins/create/create_generic.py # server_addon/houdini/client/ayon_houdini/plugins/publish/collect_houdini_batch_families.py # server_addon/houdini/client/ayon_houdini/plugins/publish/extract_rop.py
Hey! We have redesigned the LabsKarma node for rendering according to our studio requirements. it's really a good one! If the ayon publishing part was added to it. The renders can be published without any hiccups. the current ayon creating karma node maynot be a best solution as its creating in "out" it doesn't make much sense TBH when artist have to set karma properties in LOP. I know lot of studios have there own file cache and render HDAs. I love the concept of adding AYON parameters into the node and publish through that node. Lot of studios are migrating to solaris Karma mostly due to materialx support. I think it will be beneficial. Do checkout the LabsKarma ayon can add parameters and publish through it, it would be awesome! |
Hi @MustafaJafar @BigRoy @fabiaserra ! I was just exploring this PR and I have some feedback on this: adding a fetch is really a cool idea, maybe we should whitelist it for the publishing |
…ancement/houdini_generic_publish # Conflicts: # server_addon/houdini/client/ayon_houdini/api/plugin.py
Will
Will it make use of |
This point actually relates more now to the separated PR #691 - but I'd say if we can go through
Not necessarily - this PR now focuses mostly on the 'generalization' of ROP nodes in Houdini to detect them as publishable. |
I've had my doubts about that - what do you think?
Ah yes - that doesn't seem great. I'm starting to hate the publisher UI ;) Anyway, hacked around it with 939ee37 |
server_addon/houdini/client/ayon_houdini/plugins/create/create_generic.py
Show resolved
Hide resolved
…_generic.py Co-authored-by: Mustafa Taher <[email protected]>
Publisher attributes specific to the instance instead of the creatorPerfect - thanks. This will be hard to fix with the design of the publisher's logic unfortunately. A single Creator can't define its properties per instance, but only for the creator. That, by design, then limits how we can generalize these for a single Creator unfortunately since it can only have the state of one. Where in this case for render instances we should be displaying different properties. There are some other major issues that I was hoping to write up - because @dee-ynput also mentioned in essence that's also a goal of this, basically figuring out what the needs are for an API and the UI to follow suit. Other things to keep in mind are: Single output path versus a ROP that may generate any amount of "linked files"
This may be somewhat resolved by exposing attributes to the user where they can configure additional files to ingest, but for complex cases the only real sensible way to do this - is to do it programmatically, because we don't want to continuously have artists need to manually set tons of files a particular ROP may output (if those are dynamic) which again may make it hard to abstract/generalize. Although for a particular ROP node we could have a A lot of this boils down to finding what is the easiest open-accessible, easy to code yet also easy to maintain API structure we can design for creators and publishing - where we hopefully can avoid as much as possible hardcoding 'specificness' in design. I had the same discussion with @fabiaserra recently where he said "we should rely less on Creators for publishing in Houdini" but then at the same time his codebase has 'wrappers' for Houdini node types that in a way are trying to define what something should publish as, what the default parm values are, etc. In essence what he's done is also creating an API for what we need to prepare something for publishing, basically designing his own "Creator" API. In his implementation it could work to e.g. have (oversimplified pseudocode): class UsdRopNode(ayon_houdini.NodeWrapper):
def on_created(self, node):
self.make_publishable(node)
self.set_parm_defaults(node)
def get_expected_files(self):
return [file1, file2]
def set_parm_defaults(self, node):
# update the parm defaults for this node
pass We're just shifting where we're putting the logic. However, being able to access some of these bits outside of Yet, as soon as that wrapper also need to expose what settings are user configurable, etc. we're basically expanding the TL;DR - finding the simplest API that can do all we want is hard, but should be the goal |
I was trying submitting to farm and it may need additional work because:
|
Thanks for the explanation. |
…ancement/houdini_generic_publish
I'll close this PR and reopen it on separated |
Changelog Description
This PR implements an idea for "lower level publishing" in Houdini. This implement Generic ROP publishing. Just create any Houdini ROP node (or custom Rop node HDA) and publish your product types from it!
This PR originally also contained a Dynamic Runtime Instance creator. That is now separated to another PR here: #691
Additional info
As part of the Ynput Houdini Workgroup sessions I developed this quick prototype to expose a way to batch publish and ingest files. Consider it more of an exploration of what's possible then a "drop it in production now" ready-to-go solution.
Explainer
Yes, this requires some explanation. Here you go.
2024-05-22.20-48-11.mov
What I forgot to add is that it currently still relies on detecting what the output files are for a ROP node based on a "file" parm that is often unique in Houdini per node. If anyone knows a way to just query the expected output files for a ROP node (similar to what PDG/TOPs seems to do I'd love to know!) but otherwise we'll just need to expand that list.
However, I also played with the idea of having "custom file list" attributes on the Node that when enabled could override the "Collector" logic and would instead use that list of files as the publish instance's files. So that e.g. one instance could also publish multiple representations. For that, @MustafaJafar did this lovely non-functional 'prototype' but it does get the idea across:
TODO
$OS
as default variant name.ayon+settings://core/publish/collect_comment_per_instance
(although not houdini-specific nor as nice?)Demo scene file
The demo scene file:
ynts_char_hero_pdg_v012.zip
Testing notes: