ViewCoordinates does not change 3D Handedness

[zalo]

Describe the bug
The ViewCoordinates docs suggest that the global handedness of the coordinate space can be changed via logging ViewCoordinates to the root entity ala:

rr.log("/", rr.ViewCoordinates.RIGHT_HAND_Y_UP, static=True)

or

rr.log("/", rr.ViewCoordinates.LEFT_HAND_Y_UP, static=True)

However, both of these settings yield exactly the same final scene coordinates, as if the coordinate definition is only changing the orientation of the root object, and not also its scale.

Expected behavior
Setting the ViewCoordinates should cause all aspects of the 3D visualization to adhere to the new coordinate system formalisms, in terms of both handedness and orientation.

Screenshots
Setting rr.ViewCoordinates.RIGHT_HAND_Y_UP

Setting rr.ViewCoordinates.LEFT_HAND_Y_UP

Desktop (please complete the following information):

• OS: Windows 11

Rerun version
0.18.2

Additional context
Working with OpenCV's coordinate systems is painful enough without having to swizzle everything back and forth for visualization 💀

[emilk]

Rerun doesn't yet support left-handed coordinate system… though I notice our docs doesn't mention that 🤦

Sorry for the confusion. I'll update the docs right away. In the meantime, this is the issue for tracking left-handed coordinate system:

• Left-handed coordinate systems #5032

Right now our ViewCoordinates are generally quite confusing. We have a ticker for improving things:

• Split ViewCoordinates component #2663

[emilk]

Docs updated in

• Document that Rerun does not support left-handed coords #7690

[nikolausWest]

Additional context
Working with OpenCV's coordinate systems is painful enough without having to swizzle everything back and forth for visualization 💀

Doesn't OpenCV use a right handed coordinate system?

[zalo]

Additional context
Working with OpenCV's coordinate systems is painful enough without having to swizzle everything back and forth for visualization 💀

Doesn't OpenCV use a right handed coordinate system?

Heheh, funny you mention that...

Shortly after making this issue, I got enough visualization in to discover that my points were being triangulated behind the cameras instead of in front of them. Eventually I pinned it down to dealing with the inverse of the position/rotation, causing my rays to converge behind the cameras, giving me mirror-universe triangulations. Inverting the transforms gave me good, right-handed universe points and now everything lines up.

So the "need" for a left-handed system was my fault after all, and rerun saved my butt for debugging 😅.

Thank you for looking at this and keeping me honest, and I look forward to using rerun more as time goes on 👍

[nikolausWest]

@zalo: Awesome to hear that you ended up solving the issue! In prior lives I've spent so many hours debugging coordinate system transforms and projections it makes me shiver thinking about it.

Btw, very unsolicited but still, my hardest won advice on this topic (other than visualize all the things): always name any transform variable in your code with the scheme a_from_b, where a and b are the names of coordinate spaces. For example, don't call it intrinsics, call it image_hom_from_camera (or similar). Adopting this scheme and sticking to it very strictly has saved me so much time.

Please do feel encouraged to keep opening issues when you find something a bit off (or even just annoying) or if there is something you think we could add that would make a difference for your use case.

[zalo]

That's great advice... so much so, that I already had the intermediate matrix named cam_to_world... but my sleep deprivation when parsing the calibration json was stronger 😄

Should any unfortunate soul find themselves in this thread after running into a similar issue (more likely than I think 💀 ), don't be fooled by the rvec/tvec arguments in the projectPoints function; here is a magic incantation to use OpenCV's rvec's, tvec's, and fisheye module to project points from the origin coordinate space to the camera-image coordinates that you got from calibration:

def rtvec_to_matrix(rvec=(0,0,0), tvec=(0,0,0)):
    "Convert rotation vector and translation vector to 4x4 matrix"
    rvec = np.asarray(rvec)
    tvec = np.asarray(tvec)

    T = np.eye(4)
    (R, jac) = cv2.Rodrigues(rvec)
    T[:3, :3] = R
    T[:3, 3] = tvec.squeeze()
    return T

def matrix_to_rtvec(matrix):
    "Convert 4x4 matrix to rotation vector and translation vector"
    (rvec, jac) = cv2.Rodrigues(matrix[:3, :3])
    tvec = matrix[:3, 3]
    return rvec, tvec

cam_to_world = rtvec_to_matrix(cam_rvec, cam_tvec)
cam_pts = (points_3d @ cam_to_world.T)[:,:3] # Put the points into the local space of the camera
image_space_points = cv2.fisheye.projectPoints(cam_pts.reshape(1, -1, 3), np.zeros((3)), np.zeros((3)), camera_matrix, dist_coeffs)[0].squeeze()

Thank you again for the visualization library to help me catch these things!