estimateLidarCameraTransform

Estimate rigid transformation from lidar sensor to camera

Syntax

tform = estimateLidarCameraTransform(ptCloudPlanes,imageCorners)

tform = estimateLidarCameraTransform(ptCloudPlanes,imageCorners,intrinsics)

[tform,errors] = estimateLidarCameraTransform(___)

[___] = estimateLidarCameraTransform(___,Name,Value)

Description

tform = estimateLidarCameraTransform(ptCloudPlanes,imageCorners) estimates the transformation between a lidar sensor and a camera using the checkerboard planes extracted from lidar sensor data and 3-D image corners of the checkerboard extracted from camera data, respectively.

tform = estimateLidarCameraTransform(ptCloudPlanes,imageCorners,intrinsics) uses the checkerboard planes extracted from a lidar sensor, 2-D or 3-D image corners of the checkerboard extracted from a camera, and the camera intrinsic parameters to estimate the transformation between the lidar sensor and the camera.

[tform,errors] = estimateLidarCameraTransform(___) returns the inaccuracy in estimating the transformation matrix errors using any combination of input arguments in previous syntaxes.

[___] = estimateLidarCameraTransform(___,Name,Value) specifies options using one or more name-value arguments in addition to any combination of arguments in previous syntaxes. For example, 'Verbose',true sets the function to display progress.

Examples

Estimate Rigid Transformation from Lidar to Camera

Estimate the rigid transformation from a lidar sensor to a camera using point clouds captured with the lidar sensor and images captured with the camera.

Load Data

Load images and point cloud data into the workspace.

imageDataPath = fullfile(toolboxdir('lidar'),'lidardata',...
    'calibration','images');
imds = imageDatastore(imageDataPath);
imageFileNames = imds.Files;

ptCloudFilePath = fullfile(toolboxdir('lidar'),'lidardata',...
    'calibration','pointClouds');
pcds = fileDatastore(ptCloudFilePath,'ReadFcn',@pcread);
ptCloudFileNames = pcds.Files;

Load camera intrinsics into the workspace.

cameraIntrinsicFile = fullfile(imageDataPath,'intrinsics.mat');
intrinsics = load(cameraIntrinsicFile).intrinsics;

Detect Board Corners

Disable warning about using a symmetric board (not recommended). In this case, using a symmetric board does not affect the results because the padding is the same along every side of the board, and the board does not rotate more than 45 degrees in the collected images.

currentState = warning;
warning("off","vision:calibrate:boardShouldBeAsymmetric")

Detect the checkerboard points.

[imagePoints,patternDims] = detectCheckerboardPoints(imageFileNames,PartialDetections=false);

Restore the original warning state.

warning(currentState)

Specify the size of the checkerboard squares in millimeters.

squareSize = 100;

Specify the padding of the board in millimeters.

padding = [100 100 100 100];

Estimate the board corners in the camera frame.

[cornersCamera,boardSize] = estimateBoardCornersCamera("checkerboard",imagePoints,...
    intrinsics,patternDims,squareSize,Padding=padding);

Preprocess the point clouds.

numPtClouds = numel(ptCloudFileNames);
ptCloudArr = repmat(pointCloud(zeros(0, 3)),numPtClouds,1);

for i = 1:numPtClouds
    ptCloud = pcread(ptCloudFileNames{i});

    % Remove ground from point cloud.
    [~,ptCloud] = segmentGroundSMRF(ptCloud,MaxWindowRadius = 8,ElevationThreshold = 0.2,ElevationScale = 1.25);

    ptCloudArr(i) = ptCloud;
end

Specify the cluster threshold to cluster each point cloud for board detection.

clusterThreshold = 0.05;

Estimate the board corners in the lidar frame.

[cornersLidar,ptCloudBoard] = estimateBoardCornersLidar(ptCloudArr,clusterThreshold,boardSize);

Calibrate and Visualize Results

[tform,errors] = estimateLidarCameraTransform(ptCloudBoard, ...
    cornersCamera,intrinsics,Lidar3DCorners=cornersLidar);

Visualize the alignment of the board corners in the first image and point cloud pair.

% Read and undistort the first image.
idxDataPair = 1;
I = imread(imageFileNames{idxDataPair});
J = undistortImage(I,intrinsics);

% Project the board corner points from the camera and lidar to the image
% for visualization.
projectedCornersCamera = projectLidarPointsOnImage(cornersCamera(:,:,idxDataPair),intrinsics,rigidtform3d);
projectedCornersLidar = projectLidarPointsOnImage(cornersLidar(:,:,idxDataPair),intrinsics,tform);

figure
imshow(J)
hold on
showShape("polygon",projectedCornersCamera,Color="red")
showShape("polygon",projectedCornersLidar,Color="yellow")

Display translation, rotation, and reprojection errors in bar graphs.

figure
bar(errors.TranslationError)
xlabel('Frame Number')
title('Translation Error (meters)')

figure
bar(errors.RotationError)
xlabel('Frame Number')
title('Rotation Error (degrees)')

figure
bar(errors.ReprojectionError)
xlabel("Frame Number")
title("Reprojection Error (pixels)")

Input Arguments

ptCloudPlanes — Segmented checkerboard planes

P-by-1 array of pointCloud objects

Segmented checkerboard planes, specified as a pointCloud object or P-by-1 array of pointCloud objects. P is the number of point clouds. Each pointCloud object must contain points that represent a checkerboard (rectangular) plane.

P must be equal for both the ptCloudPlanes and imageCorners arguments. This means that number of point clouds and number of images used for detection must also be equal.

imageCorners — Checkerboard corners extracted from camera data

4-by-2-by-P array | 4-by-3-by-P array

Checkerboard corners extracted from camera data, specified in 2-D or 3-D coordinates.

• 2-D coordinates, specified as a 4-by-2-by-P array. Each row of a channel is of the form of [x y], of a checkerboard corner extracted from the corresponding camera image. The values are in pixel coordinates

• 3-D coordinates, specified as a 4-by-3-by-P array. Each row of a channel is of the form of [x y z], of a checkerboard corner extracted from the corresponding camera image. The values are in world coordinate system.

P represents the number of camera images used for detection. P must be equal for both the ptCloudPlanes and imageCorners arguments. This means that number of point clouds and number of images used for detection must also be equal.

Note

When imageCorners value is in 2-D coordinates, you must specify the camera intrinsic parameters, intrinsics.

Data Types: single | double

intrinsics — Camera intrinsic parameters

cameraIntrinsics object

Camera intrinsics parameters, specified as a cameraIntrinsics object.

Note

When imageCorners value is in 2-D coordinates, you must specify the camera intrinsic parameter, intrinsics. When the imageCorners are in 3-D coordinates, intrinsics is an optional input.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: Verbose=true sets the function to display progress.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: 'Verbose',true sets the function to display progress.

Lidar3DCorners — Checkerboard corners in lidar frame

4-by-3-by-P array

Checkerboard corners in the lidar frame, specified as a 4-by-3-by-P array where P is the number of point clouds.

If the user specifies the checkerboard corners in the lidar frame, then the function does not calculate them internally.

Data Types: single | double

InitialTransform — Initial rigid transformation

identity transformation as a rigidtform3d object (default) | rigidtform3d object

Initial rigid transformation, specified as a rigidtform3d object.

The function assumes the rotation angle between the lidar sensor and the camera is in the range [-45 45] along each axis. For any other range of the rotation angle, use this name-value pair to specify an initial transformation to improve function accuracy.

Verbose — Display function progress

false or 0 (default) | true or 1

Display function progress, specified as a logical 0 (false) or logical 1 (true).

Data Types: logical

Output Arguments

tform — Lidar to camera rigid transformation

rigidtform3d object

Lidar to camera rigid transformation, returned as a rigidtform3d object. The returned object registers the point cloud data from a lidar sensor to the coordinate frame of a camera.

errors — Inaccuracy in the transformation matrix estimation

structure | P-element numeric array

Inaccuracy of the transformation matrix estimation, returned as a structure or P-element numeric array.

• The function returns a structure when image corners are in 3-D coordinates. The structure contains these fields.

  • RotationError — The difference between the normal angles defined by the checkerboard planes in the point clouds (lidar frame) and those in the images (camera frame). The function estimates the plane in the image using the checkerboard corner coordinates. The function returns the error values in degrees, as a P-element numeric array. P is the number of point clouds.

  • TranslationError — The difference between the centroid coordinates of checkerboard planes in the point clouds and those in the images. The function returns the error values in meters, as a P-element numeric array. P is the number of point clouds.

  If you specify camera intrinsic parameters to the function using intrinsics argument, then the structure contains this additional field.

  • ReprojectionError — The difference between the projected (transformed) centroid coordinates of the checkerboard planes from the point clouds and those in the images. The function returns the error values in pixels, as a P-element numeric array. P is the number of point clouds.

• For 2-D image corners, the function only returns the reprojection error.

References

[1] Zhou, Lipu, Zimo Li, and Michael Kaess. “Automatic Extrinsic Calibration of a Camera and a 3D LiDAR Using Line and Plane Correspondences.” 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, October 2018, 5562–69. https://doi.org/10.1109/IROS.2018.8593660.

[2] Arun, K. S., T. S. Huang, and S. D. Blostein. “Least-Squares Fitting of Two 3-D Point Sets.” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-9, no. 5 (1987): 698–700. https://doi.org/10.1109/TPAMI.1987.4767965.

Version History

Introduced in R2020b

R2025a: MATLAB Compiler support has been removed

The estimateLidarCameraTransform function is no longer supported with MATLAB^® Compiler™.

R2022b: Supports rigidtform3d objects

You can now specify the InitialTransform name-value argument as a rigidtform3d object, which uses the premultiply convention. Although you can still specify InitialTransform as a rigid3d object, this object is not recommended because it uses the postmultiply convention. For more information, see Migrate Geometric Transformations to Premultiply Convention.

When you specify InitialTransform, the estimateLidarCameraTransform function returns tform as an object of the same type. When you do not specify InitialTransform, the estimateLidarCameraTransform function now returns tform as a rigidtform3d object. Before, the function returned tform as a rigid3d object.

R2022a: Support for 2-D image corners input

You can specify the imageCorners input in 2-D coordinates. You must additionally specify the camera intrinsic parameters by using the instrinsics input to estimate the transformation matrix when image corners are 2-D.