How to get the original unrectified pixels locations of the pointcloud created by rectification which changed the image due to it?

Hi, so during creating a point cloud using calibration parameters matlab enforces you to rectify the images. My image is 720x1280 and after rectification it becomes 486 x 976. From the rectified images I can succesfully create the desirable pointcloud, but now assume that the images that callibrated between them to create the point cloud are aligned with some other camera during also a calibration process.

The rectification distorted the image for the puprose of point cloud creation and also distorted the previously done alignement with other images. For each point in the created pointcloud I know it's pixel coordiantes because the points created from the rectified images and the rectified images have the same dimensions. i.e in my case 486x976. But what if I need the correspondence to the original 720x1280 dimensions ?

I understand that even if there is a way to do so, the number of point became less, so I lost some information to eventually to build a 3d pointcloud and thats ok for me. But how can I know for example the (x,y,z) for the (1,1) (or for another point for which it exists, because as I said there are less points, so it may not exist for this or that specific point) location in the images before rectification ?

Matlab doesn't consider the case and doesn't support this functionality ?

See the comments for more clarification.

Thank you in advance.

[Image Analyst]

What's your definition of "rectification"? Also, can you attach screenshots?

[Ilya K]

@Image Analyst, by rectification I mean the matlab operation of:

[frameLeftRect, frameRightRect] = rectifyStereoImages(frameLeft, frameRight, params);

Where `params` is the parameteres got by the function `estimateCameraParameters`.

An example of images: original (720x1280), rectified (486x922) and those who the original are aligned to (also 720x1280 unsurprisingly)

original (unrectified) :

rectified:

the image with which the original is initially aligned:

If you want effictively to see the alignment and the rectification distortion, just save the images on your desktop and open them all in full screen mode and look at the differences of the projection of one onto another changing between the windows.

I should also add that my first idea was to apply rectification to the last image to distort it the same way my original images distorted (which is necessary for the point cloud creation pipeline as it is done in matlab), and that works for the color image, but I should also say that for that color image there is also a depth image which is also aligned to it and applying the rectification for the same distortion distorts the pointcloud reconstructed from the color and the depth in an undesirable way (in a bad way partially, there is bad distortion).

and the dimensions:

P.S: here I only showed the left image, but for the gray-scale there is also a right one (for the pointcloud creation of course, but one image illustrates the rectification distortion).

Assume I created the points using :

points3D = reconstructScene(disparityMap, params);

the dimensions of it are:

as you can see they are as the rectified image dimensions. But I will to have the correspondence of the pointcloud points to the original 720x1280 pixels.

I hope that clarifies all of it much more.

Answer 1

[Aishwarya]

Hi Ilya,

Based on the details provided, it appears you are seeking a method to establish the correlation between the points in a 3D point cloud, which has been generated from rectified stereo images, and the corresponding pixel coordinates in the original unrectified images.

Here are a few insights that can help with the query:

• As rectification is performed using the “rectifyStereoImages” function, MATLAB internally computes a mapping (undistortion and rectification) that transforms points from original images to the rectified images. However, there does not appear to be any parameter that can give information about the mapping.
• One way to get the correspondence between a rectified image and its original image is by performing feature matching between, say, the rectified right image and the original right image.
• Alternatively, you could utilize the 3D world points that have been reconstructed and project them back onto the original image plane using the camera's projection matrices.

You can refer to the “rectifyStereoImages” documentation for more information: https://www.mathworks.com/help/vision/ref/rectifystereoimages.html

I hope this clarifies your query!