4.7 million prims usdstage::load is slow even if multithreaded

[pberto]

So I have a scene with ~4.7M prims out of which just 0.5M are instances.

The time to create the stage is 40 seconds. I see all cores spinning so obviously USD is trying to read it in parallel as UsdStage::Load should. Now, if I dump the whole scene to a NSI file and load it with just 1 single core , it takes also 40 seconds. So it seems there should be space for USD to do a better job at doing that multi-thread reading ;-)

The scene is the activision caldera scene with full geom on all the assets prior to over "st_main".

Also I switched off the layers with player data, so you could replicate this easily. Again the problem here is that the parallelized UsdStage::Load does not seem very efficient albeit using all cores. Would be nice to understand why. I can send you the .usda file if needed.

[jesschimein]

Filed as internal issue #USD-10263

[spiffmon]

Hey congrats on NSI being so awesome, @pberto ! USD has to do quite alot of work that I don't expect a flat NSI does. A good chunk of it goes into the fact that USD prepares for authoring/editing in addition to just extracting the data. A project we are hoping to get to in the next year is allowing clients to open a UsdStage without (or deferring) the cost of computing and building all of those editing dependencies, which I expect should help here, though we haven't yet done measurements to guide our expectations.

[pberto]

I prefer without than defer @spiffmon :-)

Actually thanks to this test I think NSI reading will become multi-threaded in the future (to the extent it is possible).

Here some stats just using USD:

* Time to load USD file: 0.00 seconds
* Time to create the USD stage: 33.68 seconds
* Time to traverse and load the USD stage: 5.79 seconds
* Time to resolve instances and create prototypes: 0.23 seconds
* Total time taken: 39.70 seconds
* Number of actual instances: 494896
* Total number of prims: 4716977
* Peak RAM: 12GB

This is the .usda I used:

#usda 1.0
(
    "usd file created with the example script from the caldera data set."
    endTimeCode = 900
    framesPerSecond = 30
    metersPerUnit = 0.0254
    startTimeCode = 0
    subLayers = [
        @./map_source/mp_wz_island.usd@,
        @./layers/cameras.usd@,
    ]
    timeCodesPerSecond = 30
    upAxis = "Z"
)

over "world"
{
    over "mp_wz_island"
    {
        over "mp_wz_island_paths"
        {
            over "mp_wz_island_geo"
            {
                over "map_phosphate_mine" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_p" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_infil_ch3" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_gulags" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_d" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_c" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_exfil_ch3" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_e" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_f" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_h" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_j" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_k" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_l" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_o" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_ruins" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "mv_intel" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_vista" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_village" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_i" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_b" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_airstrip" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_docks" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "map_tile_g" (
                    variants = {
                        string districtLod = "full"
                    }
                )
                {
                }

                over "st_main"
                {
                    over "st_b" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_d" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_c" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_f" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_e" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_h" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_g" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_j" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_i" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_l" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_k" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_n" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_p" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }

                    over "st_o" (
                        variants = {
                            string districtLod = "proxy"
                        }
                    )
                    {
                    }
                }

                over "map_beachhead" (
                    variants = {
                        string districtLod = "proxy"
                    }
                )
                {
                }

                over "map_subpen" (
                    variants = {
                        string districtLod = "proxy"
                    }
                )
                {
                }

                over "map_tile_n" (
                    variants = {
                        string districtLod = "proxy"
                    }
                )
                {
                }

                over "map_agricultural_center" (
                    variants = {
                        string districtLod = "proxy"
                    }
                )
                {
                }

                over "map_airfield" (
                    variants = {
                        string districtLod = "proxy"
                    }
                )
                {
                }

                over "map_arsenal" (
                    variants = {
                        string districtLod = "proxy"
                    }
                )
                {
                }

                over "map_capital" (
                    variants = {
                        string districtLod = "proxy"
                    }
                )
                {
                }

                over "map_caldera" (
                    variants = {
                        string districtLod = "proxy"
                    }
                )
                {
                }
            }
        }
    }
}

[pberto]

Btw, for completeness sake 🍶, here's the stats with everything set at full (and no player data) -- basically replace the word proxy with full in the above .usda:

* Time to load USD file: 0.00 seconds
* Time to create the USD stage: 99.35 seconds
* Time to traverse and load the USD stage: 32.61 seconds
* Time to resolve instances and create prototypes: 0.74 seconds
* Total time taken: 132.70 seconds
* Number of actual instances: 1248575
* Total number of prims: 12941414
* Peak RAM: 20GB

[pberto]

Any feedback here? spending 100 seconds to build the stage is a bit scary hehe. I'd like to know whats the plan in the medium term.