RGB and Depth sensors

[vikashplus]

Opening a thread to discuss if it's worth adding RGB, Depth, point cloud sensors as MuJoCo sensors.

Con

• GPU based rending passes will be required to maintain speed as CPU based passed will be quite slow

Pro

• Visual sensing is currently left for the users to handle. This will consolidate this gap. An enable/disable flag can be used to prevent unnecessary computation.
• We can pre allocate memory for these values in the data.sensordata array.
• Sim can support dynamic visual screen that updates based on camera (simulated or real) inputs.
• Sim can support dynamic point clouds that update based on camera (simulated or real) inputs.

[yuvaltassa]

Thanks Vikash.

I'd like to add a few points.

1. Pointclouds and depth maps are in principle already supported by rangefinder sensors which are cheap and work on the CPU. One easy-to-add sensor type would be a rangefinder array, that would save the user from having to manually or procedurally create lots of rangefinders. What would you think about that possibility?
2. RBG are bytes, and it would be very wasteful to cast them to doubles on data.sensordata. One could pack 3 bytes into a double... but that's rather ugly. One could imagine adding a data.rgbdata field?
3. One of the things that is currently missing regarding visual sensors is simply the specification. For example there no place to specify the resolution of cameras, or RGB/depth/grayscale. Possibly most important though is the rate. Physics is usually run a much higher rate than camera rendering. I think this is the main reason to not render pixels from within the physics loop. One fairly easy option in this regard is to add camera sensors with all the appropriate specs and memory allocation on mjData, and give the user a function that they can choose to call after an mj_step() that will fill in the visual data according to the specs.

[vikashplus]

1. I like this suggestion. I'm interested in real2sim where I'd like to bring in real sensor scans to the sim. Will it be possible to update the sensor visualization options to enable this?
2. Yes, we might need some thought here. Not only just rgb, we should also consider depth, segmentation masks, etc. Options -

• use data.sensordata but provide typecasting functionality
• create data.3Dsensordata, data.2Dsensordata, etc. This can help keep the data in contiguous arrays and avoid mixed datatypes from different sensors

3. I'm a big supporter of this suggestion. I see big wins here, especially in memory pre-allocation.

[beneisner]

Coincidentally, have also recently found myself wanting an out-of-the-box depth / point cloud sensor (and have been playing around trying to get high-accuracy depth scans from OpenGL rendering, with pretty poor results). So a rangefinder array would be quite welcome, as building a full depth image from rangefinder sensors seems cumbersome.

[yuvaltassa]

@beneisner, @vikashplus,

How would you parametrize a rangefinder array? What do you think of:

<rangefinderarray fov="x y" size="n m"/>

Here x and y would be angles respecting <compiler angle>, where the array of rangefinders is an angular grid of n-by-m elements emanating from the sensor origin, sweeping around the y-axis at angles [-x x] and around the x-axis at angles [-y y]. x = y = 0 points straight down the negative z-axis, like in cameras. x == 0 iff n == 1 and y == 0 iff m == 1, with errors thrown otherwise.

One could also consider a boolean flag parallel="false/true" (defaults to false), which means the array is composed of parallel rays and x and y are interpreted in units of length (so the array is a rectangle in space). Maybe "orthographic" is better than "parallel"?

WDYT?

[yuvaltassa]

What about the parallel/orthographic flag? Is that useful? Do such arrays exist?

[beneisner]

I think an orthographic projection would be useful (although not to me specifically, so far, as I've only used the pinhole model). However, several prominent works make use of orthographic depth projections (Google Brain's Transporter Nets and MIRA), although they seem to use pybullet for everything thus far (including rendering).

[beneisner]

And yeah I think the terminology "orthographic" is more precise from the graphics/rendering world (although I am no expert).

[vikashplus]

I have no particular experience with different kinds of projections.

[aftersomemath]

Neglecting occlusions, it is possible to transform orthographically projected points to perspective-projected points and vice-versa so long as you have the intrinsics of the two sensors.

I believe I have seen examples of 3D scanners with arrays of rangefinders which would then yield an orthographically projected point cloud, but I am struggling to find a specific example right now.

[aftersomemath]

Hi, I have been using MuJoCo for simulating cameras and point clouds for several months. The results have been good for my purposes. @beneisner @vikashplus If you are interested I can put together a short example.

As mentioned before. Depth point clouds can be extracted by from OpenGl's Z-buffer (depth image) that is produced through rasterization. I have had good results for scenes between 0.05 and 10 meters and get a nice point cloud once everything is scaled by the inverse of the camera intrinsics. When you combine this with MuJoCo's segmented rendering you get a labeled point cloud.

OpenGL uses logarithmic depth transform which introduces limitations on depth accuracy. I am not sure if it is possible to change OpenGL to use a linear depth buffer with perspective projection. This stackoverflow suggests it is possible with orthographic projection.

Will an array of rangefinders be fast compared to GPU based depth rendering? GPU's are purpose built for rasterization and one of the main outputs is a depth array.

[aftersomemath]

I found a small error in the perpendicular plane test that changed the results slightly (range is only 0.1 to 10 meters). I edited my post above to reflect the new numbers.

When the plane is tilted 45 degrees, I get a maximum error of ~1.5mm with average of -680 micrometers over a range of 0.1 to 9.5 meters (still just testing the center pixel). However, I currently have added an unexplained offset of -0.5 pixels added to the intrinsics matrix that makes the results 3-4x more accurate. I'll be putting this code on a separate branch for now here.

I think I must be calculating the intrinsics matrix for the OpenGL parameters incorrectly, especially since the unexplained offset makes a big difference.

[gywhitel]

Thanks. Inspired by @aftersomemath, I also simulated a RGBD camera in C++. Check it out here gywhitel/mujoco_RGBD, if interested.

[aftersomemath]

The source of the unexpectedly high (millimeter) OpenGL depth error with a titled plane was camera intrinsics. Once fixed, the error goes from millimeters to 10's-100's of micrometers. (znear=0.1, zfar=25, model extent=1, plane midpoint varying from 0-14 meters).

Additionally, hacking MuJoCo to use a 32 bit float reversed Z buffer instead of 24 bit integer buffer results in micrometers of error.

In the plot below blue is with default MuJoCo. Orange is with the 32 bit float reversed Z buffer.

X axis is the midpoint distance of the randomly tilted plane that fills the filed of view. The mean, std dev, min, and max are calculated per frame (per midpoint distance).

Details

I will clean up the depth test script soon and make a PR to allow users to turn on the 32 bit z buffer and reversed-Z rendering. Code for now is here and here.

I turned to optimization to find the OpenGL camera intrinsics precisely. Based on the results, I haven't been able to find a nice closed form for these parameters, and I suspect it is because the precise center of a pixel maybe implementation dependent. Doing this brought the error in the depth of a tilted plane back down from the millimeters to the micrometers (previously only a parallel plane was accurate to micrometers). Testing on a different randomized set of tilted planes reveals the estimated intrinsics are not overfit.

MuJoCo is using a 24 bit integer Z buffer which internally represents [znear, zfar] as [-1, 1] (OpenGL then linearly scales this to [0,1]). I have a hacked version with a 32 bit float reversed Z buffer which represents [znear, zfar] as [1, 0]. The extra bits make a big difference, but the reversed-Z trick is well known in the graphics community to make a difference too, see here and here.

[beneisner]

@aftersomemath Thanks for pushing for the improvements merged in #978, seems like a great addition. Will try out to see if this improves ground-truth accuracy for us. Does that PR have any impact on the inaccuracy of the intrinsics mentioned above? Or is that still unexplained?

[aftersomemath]

Thanks! Hope it helps.

The difference between what I expect the intrinsics to be and what the intrinsics used by OpenGL are is still unexplained, I'm not sure if it is my misunderstanding of the OpenGL documentation or if the direction associated with the exact center of a pixel is actually driver defined. I was planning to do a deeper investigation into this in the future especially with respect to the newly introduced cam_projection sensor.

The focal length is very close to being correct, however optimization consistently finds that the priniciple point that you would expect is incorrect by almost exactly 1/8 of a pixel in the x direction and -3/8 of a pixel in the y direction. This does not seem like an accident. You can see how I am calculating the expected OpenGL intrinsics here. At first glance, it appears the new cam_projection sensor does not account for this shift, see here. Of course it is arguable that the cam_projection sensor should be invariant to OpenGL implementation details.

[michael-projectx]

Reviving this thread to see if there is any plan to have a built in depth sensor in mujoco. I've looked at the various examples presented here and I'm happy to adapt one to my needs, but if there's an exist branch that I could check out, I'm happy to give that a try as well.

[yuvaltassa]

There was supposed to be a rangefinder array sensor, but I think that got stuck for some reason. @quagla is there a status update?

[giorgionicola]

Hi, is there any update on the topic? Any official solution?

[yuvaltassa]

it's now very easy to create a rangefinder array using replicate

[yuvaltassa]

Here you go

<mujoco model="rangefinder array">
  <worldbody>
    <geom name="floor" type="plane" size="2 2 .01"/>
    <geom size=".4"/>
    <light pos="0 0 3"/>
    <body name="1" pos="0 0 1">
      <joint axis="0 1 0"/>
      <geom type="box" size=".05 .05 .05" pos=".1 0 0"/>
      <replicate count="4" euler="0 4 0">   
        <replicate count="4" euler="0 0 -4">
          <site name="rf" pos=".16 0 0" zaxis="1 0 0"/>
        </replicate>
      </replicate>
    </body>
  </worldbody>

  <sensor>
    <rangefinder site="rf"/>
  </sensor>
</mujoco>

[anahrendra]

will there be any PR for specific implementation example for this in any robot?
looking forward to it!

[Josh00-Lu]

Are there any methods to exclude certain geoms and let the ray pass through them, instead of setting their alpha(s) to 0?

[yuvaltassa]

@Josh00-Lu
Currently there is no way to do this with built in sensors, but the ray casting functions provide this API if called directly:
https://mujoco.readthedocs.io/en/latest/APIreference/APIfunctions.html#ray-casting

[giorgionicola]

Hi I've tested using rangefinders and I have found some issues...

Here is the model I am using.

`

<option gravity="0 0 0"/>
<extension>
	<plugin plugin="mujoco.elasticity.shell"/>
</extension>
<worldbody>
	<body name="end_effector" mocap="true" pos="0 0 0">
		<body name="camera_body" pos="0 0 0.5" euler="180 0 0">
			 <replicate count="10" euler="0 4 0">
				<replicate count="10" sep="-" euler="4 0 0">
				  <site name="rf" pos="0 0 0" zaxis="0 0 1"/>
				</replicate>
			 </replicate>
		</body>
	</body>
	<body name="pin" pos="0 0 0.2">
		<flexcomp name="flag" type="grid" dim="2" count="11 21 1" spacing="0.05 0.05 1" mass="0.2" pos="0 0 0" radius="0.001">
			<edge equality="true" damping="0.01"/>
			<plugin plugin="mujoco.elasticity.shell">
				<config key="thickness" value="1e-3"/>
				<config key="poisson" value="0"/>
				<config key="young" value="3e1"/>
			</plugin>
		</flexcomp>
	</body>
	<body name="plate" pos="0 0 -0.01">
		<geom type="box" size="0.5 0.5 0.01"/>
	</body>
</worldbody>
<sensor>
	<rangefinder name="camera" site="rf"/>
</sensor>

`

1. Flexcomp seems to be transparent to the rangefinder as shown in the picture.
   
   The top element is the flexcomp and all the rays pass through it without detecting it.

2. I am not sure about the orientation of the sensors after the replicate (but it might be correct), given that model I would imagine the intersection between the rangefinders rays and the plate would a "squared grid" not evenly spaced, instead this is the result.
   
   while the "left side" of such grid of the seems to be straight the "bottom side" is curved. Might it be that the replicate applies incremental rotations on moving frames rather than on the original fixed frame?

3. Loading the model is very slow if you try to achieve standard resolutions for depth cameras such as 512x512. It might be faster to create a script that writes the XML file with the right number of rangefinder in the right orientation rather than using this macro..

[yuvaltassa]

Just reading this now, since you hid it so well 🙂

1. Sounds like a bug. Please open a new issue with a minimal model.
2. Yes, that is correct. I suppose replicate might not be suited for your needs.
3. Can you please post a minimal model that is slow to load (in a new issue) so that we can figure out where the slowness is coming from?

[yzwfromk]

Hello,

Recently I tried to mimic LiDAR using replicate of rangefinder sensors. However, I found a weird issue while using replicate as below:

As you can see in the screenshot, It gives me error.
If I change the number '11' to '9' or '111' or whatever number other than 2 digits number, it works without any error.
So the thing is, when I create 2 digits number-by-1-digit number (or 3 digit number), it works, but for 2 digits number-by-2 digits number, doesn't work.

Additionally, the joint actuators declared under replicate in XML file are not recognized by Mujoco. For example,

self.model = mujoco.MjModel.from_xml_path(path_to_xml_out)
self.data = mujoco.MjData(self.model)
data.actuator_length (could be different methods though)

In this code, data.actuator_length does give me only the actuators declared before the use of replicate but not the full list.

[yuvaltassa]

It seems like you are reporting two seperate bugs in a mostly unrelated discussion?

Please open two new issues with minimal reproducers and we will take a look.

Thanks