Add visualization schema proposal

[nvmkuruc]

Description of Proposal

The proposes introducing a shared schema domain to collect and standardize behaviors of primitives designed for nondiegetic visualization.

https://github.com/NVIDIA-Omniverse/USD-proposals/blob/vprim/proposals/vprim/README.md

Supporting Materials

Contributing

• I agree to and accept the Supplemental Terms.

[spiffmon]

There's alot I like here, and would like to see some rendering-folk perspectives on it.

Also:

• Would Viz prims be minimally UsdGeomXformable? Or they just refer/constrain to Xformable or BOundable prims?
• Formatting-wise, if you use text-fill or add linebreaks yourself within paragraphs, it allows more precise commenting in the gitHub UI.

[spiffmon]

Turn this into a reference to the other proposal?

[spiffmon]

One salient thing to note about this partitioning is the effect it may or may not have on usefully bounding a scene, in typical usage. Wrt lights, the non-boundable lights are typically aggregated somewhere off to the side of the "primary 3D scene hierarchy". If non-boundable non-diegetic prims will live alongside models/primitives they are "related to" then we could commonly wind up with. non-boundable scenes.

[nvmkuruc]

👍

[spiffmon]

Is a Billboard nothing more than a textured Plane inside a VizScope that anchors to a partial scene coordinate system but is screen facing? I.e. is it a concern fully handled by VizScope (which I expected to see here in this list)?

[nvmkuruc]

It could be for simple uses. Some users want billboards to be vectorized though like points and curves. A billboard cloud might warrant its own distinct definition.

Look for a revision to this document that makes VizScope more prominent shortly.

[spiffmon]

You mentioned above that formalizing Viz geometry helps us clearly demarcate what should get rendered in a "post physically-based renderer" pass, and that's compelling and something to follow up on with Hydra folks. Thinking about the example non-slicing effects presented here, we can't simply say non-diegetic implies "overlay", as a silhouetted object that is partially obscured by other 3D geometry doesn't want a full silhouette rendered. But the MacBeth example does want to be an overlay (I think?). Is the determination of whether a Viz rendering "Style" leverages the depth buffer of the 3D scene tied to whether it is screen-space defined/anchored or not?

[nvmkuruc]

rendered in a "post physically-based renderer" pass

Or a physically based renderer that handles these objects specially.

[tcauchois]

I guess there's a philosophical question; practically, we'll implement overlay and silhouette as image-plane operations because we don't want them to participate in rendering and after rendering is done image-plane is the best way to express things. Any kind of image operation can reasonably expect to get a certain amount of data, e.g. color, depth, id, etc if needed. I think silhouette can be done as an image operation as an edge-detect convolution on primId? And overlay can of course be done trivially as a primId match.

Internally we do some more complicated things where we render additional 3d geometry with custom compositing, e.g. drawing rotation widgets on top as transparent if they fail the depth test and as opaque if they pass the depth test. I don't know if we want to try to model that stuff here. In games that's something that can be modelled by materials, but since that kind of thing isn't physically based it's not captured by UsdShade. It's certainly useful for those kinds of prims to identify that they don't participate in light transport, at least, and I could imagine having hydra support for rendering 3d prims into the scene after the primary render (e.g. by running Viz prims through Storm and comping them in).

[nvmkuruc]

Everything you said makes sense to me Tom-- I'd expect that most styles would be implemented as image shaders that way.

I suspect there's probably going to eventually be a need for shaders to support more advanced effects, but I also suspect that there's a lot of concepts that benefit from a simplified initial specification. I'm hoping that we could start there.

[spiffmon]

Actually, visibility has not been pulled out of Imageable. There is a note in the class dox that we intended to do that at some point, but doing so while satisfying our goals for doing so would introduce a large, breaking change for any scenes with authored visibility, requiring fixbrokenpixarschemas to update...

[nvmkuruc]

Thanks for clarifying!

[spiffmon]

Wouldn't it be the Viz domain rather than the Vprim domain?

[nvmkuruc]

Yes!

[spiffmon]

This one somewhat throws my thoughts about the proposal into disarray... it doesn't submit to the same clean "let a rasterizer handle it in a post-pass", and it also goes against the final section below, as a fully formed material specification is required to capture light and reflections...

[nvmkuruc]

Look for some clarifications in a revision shortly. My hope would be that we could establish some set of default behavior for objects that works for most visualizations and consider if additional schemas or configurations are needed to bridge the gaps. (It's possible the set of effects that can be described using schemas without a material network ends up being too narrow.)

[tcauchois]

It does seem like there are a few entangled concepts here:
1.) Objects whose position or size are dependent on the camera.
2.) Objects that don't participate in light transport.
3.) Possibly redundant with 1 & 2, but stylizations done in the image plane. Objects in categories 1 and 2 can be placed in the physical scene, if not in USD then at least in the renderer, and I suppose a silhouette could be constructed in the 3d scene, but that wouldn't be my choice for implementation.

If we handle concerns 1 & 2 separately, having screen-constrained primitives that participate in light transport is easy.

[nvmkuruc]

Agreed 100%-- I'm working on a revision as we speak that tries to formalize some of the ideas and came to the conclusion that it's better to keep some of these concerns separate.

I've worked through "anchors" right now which I'm describing as a "nondiegetic break in a single diegetic space". That decouples it from any opinions about light transport.

[meshula]

using my few functional brain cells to parse "nondiegetic break in a single diegetic space";;;;;

"nondiegetic break in a single diegetic space"

means

"a thing on the stage that is not part of the world represented by the stage". If that's correct, could your amendment parenthetically add the explanation for people who need more sleep?

[nvmkuruc]

Thanks Nick. I'll try to clarify that statement more in my revisions. The original idea of this document assumed that all nondiegetic effects were nonphysical and therefore should be excluded from lighting. Talking to Dhruv and others, more narrow use cases emerged where it makes sense to have physically based materials and geometry constrained to view orientation.

So the revision for "anchoring" will introduce no constraints on lighting which aligns with Tom's observation that a few assumptions and usages were entangled and could be decoupled.

[nvmkuruc]

There's alot I like here, and would like to see some rendering-folk perspectives on it.

Also:

• Would Viz prims be minimally UsdGeomXformable? Or they just refer/constrain to Xformable or BOundable prims?
• Formatting-wise, if you use text-fill or add linebreaks yourself within paragraphs, it allows more precise commenting in the gitHub UI.

I think Vprims would be Xformable yes.

[tcauchois]

I understand the concerns about schema slicing, but I also think that this approach will require (in the long term) a ton of duplicate schema definitions for everything in the Gprim hierarchy, which I don't love.

At the hydra level, I'd want to implement these with the same usdImaging and hydra code and set appropriate flags for xform computation and lighting participation based on Vprim/Gprim; I think this is relatively easy to express in hydra, and would save a lot of duplicated code that would be a pretty significant maintenance burden. If this necessitated duplicated usdImaging code, that would worry me a lot.

[nvmkuruc]

The intent is definitely not to duplicate the Gprim hierarchy. To me, the main grey area (with the current hierarchy) is how to handle screen dependent points and curves. With curves, there are already multiple schema definitions and they're already partitioned between those intended for production rendering (BasisCurves) and annotation and rigging purposes (HermiteCurves and NurbsCurves).

However, that doesn't necessarily mean that Vprim is necessarily the right answer to that specific problem. Without introducing a higher level Vprim class, you could disambiguate curves that are "wires" and "tubes/ribbons". Wires without widths would be "intrinsically boundable" by their points and a "styling API" could be used to give wires nondiegetic effects like pixel widths. This would (perhaps narrowly) avoid a "slice" (like interpreting a single pixel width as a meter width) because consumers unaware of or unable to process styles would simply process the curve as an infinitely thin wire. I'd argue that the disambiguation between "wires" and "non-wires" would be best done at the schema level (ie-- Wires, but presence of widths and normals could be used as well. (I suspect that if "wire-ness" is not specified at the schema level, validators will flag curves authored without widths as errors which is why I'd lean that way.)

[tcauchois]

A note on implementation thoughts; I left a bunch of comments on the Autodesk billboards proposal along these lines:

My preferred way to do screen/camera anchoring is to augment the hydra "xform" definition to say that a transform is rooted in view space. This matches what GlfSimpleLight does for camera lights, and is consistent with the other flag we have in the hydra "xform" definition: resetXformStack, which can be thought of as saying that a transform is rooted at the origin of world space. The idea is that these xforms would flow to the renderer, and Storm (for example) could turn a view-relative transform into a world-relative transform in the shader, meaning that we don't need to recompute the transforms for each viewer in a multi-viewer app.

If we want to separate position anchoring, rotation anchoring, and scale anchoring, it's a bit more complicated but you can say (for example) that a matrix is translate-anchored to the camera and just compensate the translation as above.

I think it is cool and useful to let transforms below one of these camera-anchoring operations compose, since the composed transform would be (implied inverse view xform) * parent xform * child xform, and the child composed transform just inherits the flag indicating the implied inverse view xform.

This is all consistent with VizScreenAnchor being a scoping mechanism, and would potentially be with an xformOp as well.

Any kind of screen space positioning, scaling, styling is perfectly xformable and boundable in the renderer, of course, but absent a GfFrustum for context the UsdGeomXformCache/UsdGeomBboxCache become meaningless at some point in the tree, so for non-hydra consumers I suppose it's important to flag when that becomes the case. I think if XformCache/BboxCache had an option to take a GfFrustum, it could replicate the compensation that hydra is doing, and that would be a useful utility to provide.

[nvmkuruc]

I think consumers like hydra that can guarantee they have a single frustum have a lot of flexibility with how this gets implemented.

In my latest updates, I'm suggesting that transforms and bounds might want to be evaluated with respect to anchoring points-- to most prims that would be the pseudo root and nothing would change, but you'd be able to identify the anchoring point and use whatever extrinsic context was needed (resolution, frustum, etc.).