Can we use multiprocessing over multiple cameras?

[keiohta]

Hi author, thanks for supporting this great simulator!

I have a question related to using multiprocessing when rendering scenes. Since the most time-consuming part of my application is taking pictures using multiple cameras with different positions and orientations, I would like to speed up this part using multiprocessing. However, when I try to do that, I encounter RuntimeError: vk::Device::createGraphicsPipelineUnique: ErrorOutOfDeviceMemory. So, could you take a look at the following code and hopefully share some workarounds?

import multiprocessing

import numpy as np
import sapien.core as sapien
from sapien.utils.viewer import Viewer
from transforms3d.euler import mat2euler


def init_cam(scene, cam_pos):
    # just follow the tutorial
    near, far = 0.1, 100
    width, height = 640, 480
    camera_mount_actor = scene.create_actor_builder().build_kinematic()
    camera = scene.add_mounted_camera(
        name="camera",
        actor=camera_mount_actor,
        pose=sapien.Pose(),
        width=width,
        height=height,
        fovx=np.deg2rad(35),
        fovy=np.deg2rad(35),
        near=near,
        far=far)

    cam_pos = np.array(cam_pos)
    forward = -cam_pos / np.linalg.norm(cam_pos)
    left = np.cross([0, 0, 1], forward)
    left = left / np.linalg.norm(left)
    up = np.cross(forward, left)
    mat44 = np.eye(4)
    mat44[:3, :3] = np.stack([forward, left, up], axis=1)
    mat44[:3, 3] = cam_pos
    camera_mount_actor.set_pose(sapien.Pose.from_transformation_matrix(mat44))

    scene.step()  # make everything set
    scene.update_render()
    return camera


def take_picture_multiprocessing(camera):
    camera.take_picture()


def main():
    engine = sapien.Engine()
    renderer = sapien.VulkanRenderer()
    engine.set_renderer(renderer)

    scene = engine.create_scene()
    cameras = [init_cam(scene, cam_pos) for cam_pos in [[-2, 0, 0], [-1, 1, 0], [-1, -1, 0]]]

    loader = scene.create_urdf_loader()
    loader.load("simple.urdf")

    tasks = []
    for camera in cameras:
        tasks.append(multiprocessing.Process(target=take_picture_multiprocessing,
                                             args=(camera,)))

    for task in tasks:
        task.start()

    for task in tasks:
        task.join()


if __name__ == "__main__":
    main()

The result is as follows. You can see that RuntimeError: vk::Device::createGraphicsPipelineUnique: ErrorOutOfDeviceMemory occurs when taking images using camera:

$ python test_mp_cameras.py
[2022-02-10 08:30:05.498] [SAPIEN] [warning] Mass or inertia contains very small number, this is not allowed. Mass will be set to 1e-6 and inertia will be set to 1e-8 for stability. Actor: 
[2022-02-10 08:30:05.498] [SAPIEN] [warning] Current camera implementation does not support non-squarepixels, and fovy will be used. Set fovx to 0 to suppress this warning
[2022-02-10 08:30:06.029] [SAPIEN] [warning] Mass or inertia contains very small number, this is not allowed. Mass will be set to 1e-6 and inertia will be set to 1e-8 for stability. Actor: 
[2022-02-10 08:30:06.029] [SAPIEN] [warning] Current camera implementation does not support non-squarepixels, and fovy will be used. Set fovx to 0 to suppress this warning
[2022-02-10 08:30:06.036] [SAPIEN] [warning] Mass or inertia contains very small number, this is not allowed. Mass will be set to 1e-6 and inertia will be set to 1e-8 for stability. Actor: 
[2022-02-10 08:30:06.036] [SAPIEN] [warning] Current camera implementation does not support non-squarepixels, and fovy will be used. Set fovx to 0 to suppress this warning
[2022-02-10 08:30:06.043] [SAPIEN] [error] Cylinder visual is not supported. Replaced with a capsule
[2022-02-10 08:30:06.044] [SAPIEN] [warning] Mass or inertia contains very small number, this is not allowed. Mass will be set to 1e-6 and inertia will be set to 1e-8 for stability. Link: base_link
Process Process-1:
Traceback (most recent call last):
  File "/home/ohta/anaconda3/envs/sapien/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/ohta/anaconda3/envs/sapien/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ohta/workspace/robotics/efficient_manip/test_mp_cameras.py", line 64, in take_picture_multiprocessing
    camera.take_picture()
RuntimeError: vk::Device::createGraphicsPipelineUnique: ErrorOutOfDeviceMemory
Process Process-2:
Traceback (most recent call last):
  File "/home/ohta/anaconda3/envs/sapien/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/ohta/anaconda3/envs/sapien/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ohta/workspace/robotics/efficient_manip/test_mp_cameras.py", line 64, in take_picture_multiprocessing
    camera.take_picture()
RuntimeError: vk::Device::createGraphicsPipelineUnique: ErrorOutOfDeviceMemory
Process Process-3:
Traceback (most recent call last):
  File "/home/ohta/anaconda3/envs/sapien/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/ohta/anaconda3/envs/sapien/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ohta/workspace/robotics/efficient_manip/test_mp_cameras.py", line 64, in take_picture_multiprocessing
    camera.take_picture()
RuntimeError: vk::Device::createGraphicsPipelineUnique: ErrorOutOfDeviceMemory

Process finished with exit code 0

The urdf in the script is as follows.

$ cat simple.urdf
<?xml version="1.0"?>
<robot name="example">
  <link name="base_link">
    <visual>
      <geometry>
        <cylinder length="0.6" radius="0.2"/>
      </geometry>
    </visual>
  </link>
</robot>

Thanks in advance for your time and continual support.

[fbxiang]

To use multiprocessing with SAPIEN, you must create different engines and renderers in each process, you cannot parallelize a single function call. So what you need to do is more like parallelizing the main program.

[keiohta]

Hi @fbxiang , thanks for the quick reply! Sure, then I'll create an engine in each process and reuse it.

Just a quick question, how can we remove a loaded object in a scene? I imagine the following script, and I want to remove the loaded object in the for loop obj = loader.load(urdf) after finishing a heavy process. I tried scene.remove_actor(obj) but produced some error.

def process():
    engine = sapien.Engine()
    renderer = sapien.VulkanRenderer()
    engine.set_renderer(renderer)

    scene = engine.create_scene()

    for urdf in urdfs:
        loader = scene.create_urdf_loader()
        obj = loader.load(urdf)
        
        ... some heavy process
        
        # HERE
        scene.remove_obj(obj)

[fbxiang]

You want scene.remove_articulation since what you are removing is an articulation instead of an actor.
https://sapien.ucsd.edu/docs/2.0/tutorial/basic/create_articulations.html?highlight=remove%20articulation#remove-an-articulation

[keiohta]

Oh my bad... yeah, now my script works. Thanks a lot!