X, Y and Z co-ordinates from 3D point cloud are not accurate. Showing considerable error in X co-ordinates.

I reconstruct 3D scene using MATLAB 2014b and using point cloud I have to find out the X,Y and Z co-ordinates of white object in scene. For this, I find out centroids of white objects in Image 1 (I1)and using that values (which are in pixels off-course) I extract X, Y and Z values. But when I compared these extract values with actual values (which I measured with scale from left camera or camera 1), there was huge difference in values. Then I notice that my actual image I1 is of 480x640 while my rectified image J1 is of 442x604. Then just for trial, I found the centroid of white object in J1 (using impixelinfo) and extract the values. These values are far better then I get using I1, but still there is an avg. error of 7.31 cm in X, 0.5 cm in Y and 4.0 cm in Z. I can tolerate 0.5 cm error, so Y co-ordinate is not problem but in X and Z co-ordinates I want more accuracy (with preferable image I1 because I have code to get centroids using I1 not J1). I calibrate my stereo camera many times and still the same problem. One more point I notice that as distance along X-axis from left camera increases my accuracy increases. I can't understand, why? I'm attaching my stereoParams and excel files showing error with this.

Answer 1

[Dima Lisin]

Hi Naseeb,

First, based on what you wrote, I would guess that you may be confusing camera 1 and camera 2. According to your stereoParams object, your cameras are 9.1 cm apart. So if you are measuring locations of objects relative to camera 2 instead of relative to camera 1 that would explain most of the error in X.

Second, please keep in mind that the world coordinate system (assuming that you are using disparity() and reconstructScene() has its origin at the principal point of camera 1, which is located somewhere inside the camera casing. So that becomes hard to measure. A better test of accuracy of the reconstruction would be to measure a distance between two points in the world.

So, please check that you don't have camera 1 and camera 2 mixed up. If that does not help, could you please post a sample pair of images? Also, please clarify how exactly you are doing the reconstruction. Are you calling rectifyStereoImages followed by disparity followed by reconstructScene? If so, then of course you should be using the rectified images to look up the 3D world coordinates.

[Naseeb Gill]

Hi Dima,

Again, I calibrate my cameras and try to measure distance from either camera but error is same.

By left or camera 1, I mean, when I stand behind the camera and follow the line of sight of cameras, then my left hand indicates left camera and right hand indicate right camera. Also, even if I consider my calculations by camera 2 then also for Image 6, it may give correct x- values but for Image 13 the errors may increased more in x-values (you may see excel sheet again for image 13).

I'm reconstructing my scene calling rectifyStereoImages followed by disparity followed by reconstructScene because using triangulation error increases more.

I'm attaching my images to this comment as well as question also. I wants to give details of my measurement.

For image 6:-

top corner of white rectangular paper (away from my foot) is x = -1.9 cm; y = 3.1 cm and z = 1.7560 m when measure from left or camera 1 as I describe my left camera.

top corner (near to my foot) is x = 9.1 cm; y = 3.1 cm and z = 1.7560 m.

size of white rectangular paper is 10.5 x 7.9 cm.

For image 13:-

top corner (away from my foot) is x = -41.4 cm; y = 3.1 cm and z = 1.7560 m.

top corner (near to my foot) is x = -31.00 cm; y = 3.1 cm and z = 1.7560 m.

[Dima Lisin]

Ok, I think I know what's going on. Here's the disparity map that I got from image6:

The circled area is your sheet of paper. Notice how the edges do not look very straight. This is because a disparity map is usually not very accurate at the edges of objects. This affects your measurements. If you use the center of the paper instead, you should get much better accuracy.

If you insist on using corners, you can try decreasing the BlockSize parameter of the disparity function. That should make the object edges more accurate, but I doubt it will be accurate enough for you. Also, it will make the disparity map more noisy overall.

The alternative approach would be to use the triangulate function. Here you would not rectify the images, but instead undistort them using undistortImage function. Then you would have to match the points of interest between the two images, and then call the triangulate function to get the 3D coordinates.

[Naseeb Gill]

Hi Dima,

As you suggest to use the center of paper, I tried but error remain same. Also, I notice that if paper is on right side of left camera (i.e for Image 6) then depth shown using pointCloud is more than actual while if paper is on left side of camera (i.e for image 9) depth shown using pointCloud is less than actual, I don't know why? In X-co-ordinate, error is nearly 7-8 cm even using center of paper.

Also, as you suggest triangulate function, I tried that also and the results are high error even in Z axis also, in this case.

If you can check once more please, it may help me. Thanks.

[Dima Lisin]

Hi Naseeb,

First of all, are you sure your actual measurements are correct? Please keep in mind that the X-Y plane of your world coordinate system is the rectified image plane of camera 1. That means you do not know exactly where the origin is, and you do not know how exactly the axes are oriented in the world. So, the X-axis may not point where you think it does.

The best way to check the accuracy is to take two points in the world, and measure the distance between them. Place two sheets of paper in the scene, and measure the distance between their centers. For a more crude evaluation, you can measure the distance from the camera to the object.

Other than that, I can only give you some general tips for improving accuracy:

- Take a look at the reprojection errors of your stereo parameters using showReprojectionErrors function. There is one image pair that has a much higher error than the others. Try excluding that pair and recalibrating.

- Use a higher resolution camera. In the end your accuracy is limited by pixels.

Another thought: how do you acquire the images? If you are using two separate cameras, there is always some time lag between taking the two pictures. If you are holding the paper in your hand, it may have moved in that time, which will introduce an error in your measurement. Try using an object that you can place in the scene without having to hold it.

[Naseeb Gill]

Hi Dima,

How can I align camera co-ordinates or image co-ordinates with my real world co-ordinates. Because the X,Y and Z co-ordinates which I got, I have to use these for further processing but without knowing the origin and accuracy I can't proceed. Is there any way to find how camera co-ordinate system is look in this case and then align camera co-ordinates with world co-ordinates system?

Also while capturing images, I'm was not moving and reprojection error is less than 0.25 pixels. I can't change my cameras due to some reasons and I want as much accuracy as I can achieve using these only.

[Dima Lisin]

Hi Naseeb,

You can take a picture a of a checkerboard and use the extrinsics function. That will give you the transformation from the world coordinate system defined by the checkerboard and the camera's coordinate system.

However, I am still not clear why you need to know exactly where the origin of the camera's coordinate system is? Coordinates are always relative to something. If you need to perform measurements, then typically you are interested in distances between points. The distances do not depend on the location or orientation of your coordinate system, and you can easily measure them to verify accuracy.

About your last point, in the images you posted, you were holding a piece of paper with your hand. People's hands shake. Depending on how much time passes between taking the left and right images, your hand shaking can affect your measurement. Unless you have a "real" stereo camera with hardware triggering that takes the two images simultaneously, it is better to have an object that you can lean on something without having to hold it.

[Naseeb Gill]

Hi Dima,

Why I'm interested in origin is because I'm working on robotic arm which will pick white objects. For that I should know how much distance in X,Y and Z my arm should move to pick that object. For inverse kinematics I should know from where I'm getting these values. So, basically I'm not interested in distance between objects but distance between origin and objects.

[Dima Lisin]

Oh, I see. In that case you may want to put a checkerboard down on the table or the floor that you need to pick up the objects from, and get the extrinsics. Then you can transform everything to the coordinate system where the floor is the X-Y plane.

[Meghana Dinesh]

Hi Dima,

I had faced a similar issue (I hope you recollect the questions I had posted earlier). Anyway, I didn't yet find a reliable method to capture the checkerboard image pair (CB1) to be used by extrinsics. My camera is mounted and looks at the object (wheel). This wheel may move along x, y and z axis. Suppose it moves away from the camera on the ground plane (ideally the z-axis). How do I make sure this movement along z-axis is EXACTLY parallel to the z-axis defined by CB1?

I have tried my best to draw parallel lines on the floor, to define x and z axes; placing CB1 as accurately as possible. I am still not able to obtain a distance accuracy of 1mm (over a range of 70cm to 1.5m).

How do you suggest I proceed?

I noticed that you had told me too to calculate distance accuracy with respect to another object in the scene. This measurement in world co-ordinates would give me the euclidean distance (not the individual x, y and z distances) unless I have a well defined co-ordinate system along which I can move my object.

Naseeb,

Note that you can fix an origin by capturing a Checkerboard Image and using extrinsics function.

[Dima Lisin]

Hi Meghana,

You do not need a pair of checkerboard images to define world coordinates. You only need to capture one image with camera 1.

As to how to align coordinate systems with such precision, I don't really know how to help you. This is more of a mechanical problem.

[Meghana Dinesh]

Yes, I use Camera 1's image.

Thank you.

[Naseeb Gill]

Hi Dima,

As you suggest I used extrinsics and get rotationMatrix and translationVector. Using below code.

    I1 = imread('C:\Users\Naseeb\Desktop\scene1\left\left.png');
I2 = imread('C:\Users\Naseeb\Desktop\scene1\right\right.png');
load('C:\Users\Naseeb\Desktop\st\stereoParams.mat'); 
 im = undistortImage(I1, cameraParams);
im = undistortImage(I1, stereoParams.CameraParameters1);
[imagePoints, boardSize] = detectCheckerboardPoints(im);
squareSize = 21; % in millimeters
worldPoints = generateCheckerboardPoints(boardSize, squareSize);
[rotationMatrix, translationVector] = extrinsics(imagePoints, worldPoints, stereoParams.CameraParameters1)

I think translationVector comes out with respect to Pattern-Based Coordinate System. So, I interpolate origin with respect to this using translationVector. But, it comes 2.5 cm away and 6.4 cm below the left camera or camera 1.

Also, I used following code for my stereo Images.

    I1_undist = undistortImage(I1, stereoParams.CameraParameters1);
I2_undist = undistortImage(I2, stereoParams.CameraParameters2);
[imagePoints1, ~] = detectCheckerboardPoints(I1_undist);
[imagePoints2, ~] = detectCheckerboardPoints(I2_undist);
Points = triangulate(imagePoints1,imagePoints2,stereoParams);
TranslatePoints = zeros(size(Points));
TranslatePoints(:,1) = Points(:,1) - translationVector(1);
TranslatePoints(:,2) = Points(:,2) - translationVector(2);
TranslatePoints(:,3) = Points(:,3) - translationVector(3);
R = rotationMatrix;
RotatedPoints = TranslatePoints * (R');
TransformedPoints = RotatedPoints;

But, the problem is that, TransformedPoints are not with respect to my interpolated origin but it’s origin is somewhere in left middle of checkerboard. I still don’t know exactly from where it is measuring (origin is where). Also, it changes its value with change in image. I want to fix my origin either it at camera 1 or base of my platform as in real time I can’t put checkerboard in every image.

I'm attaching files with this images and stereoParams.

Thanks

[Dima Lisin]

Hi Naseeb,

I tried running your code, and I plotted your TrasnsformedPoints using plot3. Here's what they look like if you look from the "top":

And here's what they look like from the "side":

In other words you have a nice reconstruction in X and Y, but you have errors in Z of 2-3cm. So, what you are doing is correct, but your reconstruction has errors.

If these errors are unacceptable, then there are a few things you can do. One is to use a higher resolution camera. Another thing that may help, is to measure the size of the checkerboard square more precisely using a caliper. If you can measure it with the accuracy of .1mm rather than 1mm, that can improve the accuracy of the reconstruction.

You can also try improving the calibration accuracy by using even more calibration images. I would also try using a larger checkerboard, which may improve corner localization accuracy.

Moving the cameras a little farther apart can generally help, but probably not in this case, because your object of interest is very close to the cameras.

Also, I do not know what your stereo camera rig looks like. You absolutely have to make sure that the cameras do not move relative to each other. Even a slight shake will cause noticeable reconstruction errors.

Another thought: your checkerboard looks a bit wavy. Try making it as flat as you can. For example, glue it to a flat piece of plastic.