How to get 3D world coordinates from 2D image coordinates?

I calibrated my cameras, took stereo images, rectified them, created a disparity map and used "reconstructSecene" to create a point cloud. I then used "getpts" to get the image coordinates for my point of interest from the left image (should I use the rectified or non-rectified version of the image?). Now I want to use the image coordinates to query the 3D coordinates. What code would I use to achieve that?

Answer 1

[Qu Cao]

You should use the rectified stereo images. The disparityMap computed from disparitySGM should have the same size as your stereo image (left or right), M-by-N. The output xyzPoints of reconstructScene is of size M-by-N-by-3. So, if you get the 2-D image coordinates [u, v], you can query the corresponding 3-D point using something like:

point3d = xyzPoints(round(v), round(u), :);