How to transform 2D point coordinates after rectification?

I'm using 'estimateUncalibratedRectification' and 'rectifiStereoImages' functions to rectify two images from left and right camera.

I firstly estimate the fundamental matrix by matched points from two images. Then use the matched points and fmatrix to estimate the transformation matrixs of two images for rectification.

After the rectification, the images seems to be cropped (with less size). I try to use transformation matrix (tform1) to transform the matched points in raw image 1 to rectified image 1 but failed as it is far from where it shoud be.

Can anybody tell me how to transform these 2D point coordinates after rectification?

Best regards,

Xicheng Wang

The main code is:

[fMatrix, inliers] = estimateFundamentalMatrix(mp1,mp2,'Method','Norm8Point');

[t1, t2] = estimateUncalibratedRectification(fMatrix, mp1(inliers,:), mp2(inliers,:), size(im1));

tform1 = projective2d(t1); % transformation matrix for image1 (right)

tform2 = projective2d(t2); % transformation matrix for image2 (left)

[im1_Rect, im2_Rect] = rectifyStereoImages(im1,im2,tform1,tform2); % rectfiy the images

% Transform the points from raw image to rectified image

[mp1_Rect(:,1),mp1_Rect(:,2)] = transformPointsForward(tform1,mp1(:,1),mp1(:,2));

Answer 1

[Milan Bansal]

Hi Xicheng,

I understand that you are trying to rectify two images from left and right cameras using the "estimateUncalibratedRectification" and "rectifyStereoImages" functions in MATLAB, and you are encountering issues where the rectified images are cropped and the transformation of matched points using the estimated transformation matrix does not align as expected.

Refer to the following steps to resolve the issues with uncalibrated rectification of images:

• Use the "estimateStereoRectification" function instead of the "estimateUncalibratedRectification" function to rectify the images from the left and right cameras.
• Before rectification, ensure that the matched points (mp1 and mp2) are accurate and that there are no outliers, as they can affect the quality of the rectification.
• Try changing the "method" parameter in "estimateFundamentalMatrix" to "LMedS" or "MSAC" for calculating the fundamental matrix.
• When calling "rectifyStereoImages" function, use the 'OutputView' parameter set to 'full' to get the full images without cropping.

Please refer to the code snippet below to implement the above modifications in your code:

% Calculate fundamental matrix 
[fMatrix, inliers] = estimateFundamentalMatrix(mp1, mp2, 'Method','LMedS'); 

% Estimate the uncalibrated rectification. 
[t1, t2] = estimateStereoRectification(fMatrix, mp1(inliers,:), mp2(inliers,:), size(im1)); 

% rectfiy the images 
[im1_Rect, im2_Rect] = rectifyStereoImages(im1, im2, tform1, tform2, OutputView='full');    

% Transform the points from raw image to rectified image 
[mp1_Rect(:,1), mp1_Rect(:,2)] = transformPointsForward(tform1, mp1(:,1), mp1(:,2)); 

Please refer to the following documentation links to learn more about "estimateStereoRectification", "estimateFundamentalMatrix" and "rectifyStereoImages" functions:

• https://www.mathworks.com/help/vision/ref/estimatestereorectification.html 
• https://www.mathworks.com/help/vision/ref/estimatefundamentalmatrix.html 
• https://www.mathworks.com/help/vision/ref/rectifystereoimages.html 

Additionally, you can find a related example in the following documentation link:

https://www.mathworks.com/help/vision/ref/estimatestereorectification.html#mw_63d2ea9b-bd17-40b3-b765-7f18da728250 

Hope this helps!