Blensor pcd output laterally inverted while projecting

[agnivsen]

Issue Description:

I tried to simulate the Kinect camera using the Blensor application (from the binary release). I used the default object provided, applied an image texture on it, did some animation and rendered it into pcd files.

I want to run some computer vision algorithm on it. I load the pcd files using C++ (with PCL library). I am interested in how the pointcloud looks on the image plane. I do a perspective projection of the pointcloud using default Kinect camera parameters.

Here is when I notice that the projected image is laterally inverted, when compared with the pointcloud.

---

Here are the details:

Here are the screen grabs of the settings that I use in the Blender GUI:

How the workspace looks like.

Camera Settings.

Transformation Settings.

Settings of the Kinect Camera before scanning/rendering.

You can find a sample of the generated pointcloud in this link: Sample Pointcloud

Here's a screen shot of the PCD file when viewed in pcl_viewer (with the axes visualized):

The pointcloud in pcl viewer. This is how it should actually look like.

And here is the same pointcloud, projected in the image plane:

Projection of the pointcloud, in grayscale.

If you observe closely, you will notice that the image and the pointcloud are aligned oppositely in the horizontal direction.

I have not been able to understand the cause of the problem. Is Blensor dumping the X axis of the scan in the opposite direction? What is the problem and how can this be fixed?

---

Additional details:

If it helps, here is the code I use to project the pointcloud:

  ` pcl::PointCloud<pcl::PointXYZRGB>::iterator b1;
    std::vector<cv::Point3f> objectPoints;
    cv::Mat A(3,3,cv::DataType<double>::type);  //projection matrix
    std::vector<cv::Point2f> projectedPoints;
    std::vector<float> data;
    int i = 0;

    for (b1 = cloud->points.begin(); b1 < cloud->points.end(); b1++, i++)
    {
        if(pcl_isfinite(b1->x) && pcl_isfinite(b1->y) && pcl_isfinite(b1->z))
        {
            cv::Point3d p(b1->x,b1->y,b1->z);
            data.push_back((b1->r+b1->g+b1->b)/3);
            objectPoints.push_back(p);
        }
    }

    A.at<double>(0, 0) = cam_depth.get_px();  A.at<double>(0, 1)  = 0;                                        A.at<double>(0, 2)  = cam_depth.get_u0();
    A.at<double>(1, 0)  = 0;                                        A.at<double>(1, 1)  = cam_depth.get_py(); A.at<double>(1, 2)  = cam_depth.get_v0();
    A.at<double>(2, 0)  = 0;                                        A.at<double>(2, 1)  = 0;                                        A.at<double>(2, 2)  = 1;

    cv::Mat distCoeffs(4,1,cv::DataType<double>::type);  //distortionless projection
    distCoeffs.at<double>(0) = 0;
    distCoeffs.at<double>(1) = 0;
    distCoeffs.at<double>(2) = 0;
    distCoeffs.at<double>(3) = 0;

    cv::projectPoints(objectPoints, cv::Mat::eye(3, 3, CV_64F), cv::Mat::zeros(3,1,CV_64F), A, distCoeffs, projectedPoints);

    std::vector<cv::Point3f>::iterator it1 = objectPoints.begin() ;
    std::vector<float>::iterator it2 = data.begin() ;

    for (std::vector<cv::Point2f>::iterator it = projectedPoints.begin(); it != projectedPoints.end(); (++it) )
    {

        int x_ = round(it.base()->x);
        int y_ = round(it.base()->y);
        if((x_> 0) && (y_ > 0)&&(x_<width) && (y_ < height))
        {
            float intensity = 0.0f;
            intensity = *it2;

            color_image[y_][x_].R = intensity;  //color_image is a matrix that holda RGB values
            color_image[y_][x_].G = intensity;
            color_image[y_][x_].B = intensity;
        }
        it1++; it2++;
    }
    
    return color_image; `

This projection code works fine with other pointclouds, including the ones dumped from real Microsoft Kinect and Intel RealSense.

[mgschwan]

Is it possible that you are using an older blensor version. The most recent versions do have options to invert each axis individually to match the output coordinate system the user needs

[agnivsen]

Yes, I am using the Blensor based on Blender 2.74, possibly Blensor version 1.0.17.

Could this be the root cause?
Can we not use Blensor 1.0.18 on Ubuntu 16.04 (since the download page says 17.04+)?

Nevertheless, thanks for pointing this out. I will try out the latest version.

[mgschwan]

1.0.17 is quite old.

You can try to use 1.0.18, but there may be libraries missing. Please try the newest version first

[agnivsen]

Checked out the latest commit in Blensor repo (at commit 338ad01) and tried to repeat the same process with the new version. Here is what I get:

New camera render settings with inverted X

How the new pointcloud looks like

The new projected image

---

Summary:

With the InvX radio button checked, I managed to invert the entire system by the X-axis, but did not solve the problem. The pointcloud and its projection are still laterally inverted w.r.t each other. Moreover, flipping the entire system along its X-axis is not desired (and makes things pretty complicated, even if I try to use it as a temporary workaround).

[mgschwan]

Can you send me the blend and pcd file. Thanks Michael Am 19.03.2018 15:47, schrieb Agniva Sengupta:

…

Checked out the latest commit in Blensor repo (at commit 338ad01 [1]) and tried to repeat the same process with the new version. Here is what I get: [2] _New camera render settings with inverted X_ [3] _How the new pointcloud looks like_ [4] _The new projected image_ ------------------------- SUMMARY: With the _InvX_ radio button checked, I managed to invert the entire system by the X-axis, but did not solve the problem. The pointcloud and its projection are still laterally inverted w.r.t each other. -- You are receiving this because you commented. Reply to this email directly, view it on GitHub [5], or mute the thread [6]. * Links: ------ [1] 338ad01 [2] https://user-images.githubusercontent.com/5153445/37601978-a7288528-2b8b-11e8-8793-1570f40ff06e.png [3] https://user-images.githubusercontent.com/5153445/37602025-cada5230-2b8b-11e8-9128-5c6206ad4ee2.png [4] https://user-images.githubusercontent.com/5153445/37602064-e085752e-2b8b-11e8-8fd2-f9ada701713c.png [5] #24 (comment) [6] https://github.com/notifications/unsubscribe-auth/AAXT1Cx0KA1wOLo9LijGAh_U78FyfSYiks5tf8T-gaJpZM4StimS

[agnivsen]

The .blend file can be downloaded here:

Blender file

---

Sample pcd output files can be found here:

Sample PCD files

(You can ofcourse export from the blender file to generate the full set of pointcloud output)

[mgschwan]

Since the output of the pointcloud is correctly mapped on the error may not be in blensor itself.

I think the problem is the Z coordinate. In the normal setting the objects in front of the scanner have a negative Z coordinate. If your code projects the data onto a 2D plane without accounting for that, the output should be inverted since the object is actually behind your camera.

You should be able to fix that by inverting the Z coordinate (not X) or whatever axis is normal to you image plane.