org.openpnp.machine.reference.camera.calibration

Class CameraCalibrationUtils

java.lang.Object

org.openpnp.machine.reference.camera.calibration.CameraCalibrationUtils

public class CameraCalibrationUtils
extends java.lang.Object

Provides a set of utilities to aid in camera calibration

Field Summary

Fields

Modifier and Type	Field and Description
static int	FIX_ASPECT_RATIO
static int	FIX_PRINCIPAL_POINT
static int	FIX_RADIAL_DISTORTION_COEFFICENTS
static int	FIX_TANGENTIAL_DISTORTION_COEFFICENTS
static int	FIX_TILT
static double	sigmaThresholdForRejectingOutliers

Constructor Summary

Constructors

Constructor and Description
CameraCalibrationUtils()

Method Summary

Modifier and Type	Method and Description
static double	computeBestCameraParameters(double[][][] testPattern3dPoints, double[][][] testPatternImagePoints, double[][][] modeledImagePoints, java.util.List<java.lang.Integer> outlierPointList, double[] parameters) Computes the best, in a least squared error sense, set of camera parameters that transform the given test pattern 3D coordinates to the corresponding 2D image points.
static double	computeBestCameraParameters(double[][][] testPattern3dPoints, double[][][] testPatternImagePoints, double[][][] modeledImagePoints, java.util.List<java.lang.Integer> outlierPointList, double[] parameters, int flags) Computes the best, in a least squared error sense, set of camera parameters that transform the given test pattern 3D coordinates to the corresponding 2D image points.
static double	computeDrmsError(java.util.List<double[]> residuals) Computes the Distance Root-Mean-Squared (DRMS) error for a given set of 2D errors
static Mat	computeRectificationMatrix(Mat transformFromMachToPhyCamRefFrame, Mat vectorFromMachToPhyCamInMachRefFrame, Mat transformFromMachToVirCamRefFrame, Mat vectorFromMachToVirCamInMachRefFrame, double primaryZ) Computes the rectification matrix that converts normalized physical camera coordinates to the virtual camera coordinates
static java.util.List<double[]>	computeResidualErrors(double[][][] actual2DPoints, double[][][] modeled2DPoints) Computes the 2D error between the actual image points and the modeled image points
static java.util.List<double[]>	computeResidualErrors(double[][][] actual2DPoints, double[][][] modeled2DPoints, java.util.ArrayList<java.lang.Integer> outlierList) Computes the 2D error between the actual image points and the modeled image points
static java.util.List<double[]>	computeResidualErrors(double[][][] actual2DPoints, double[][][] modeled2DPoints, java.lang.Integer heightIndex) Computes the 2D error between the actual image points and the modeled image points
static java.util.List<double[]>	computeResidualErrors(double[][][] actual2DPoints, double[][][] modeled2DPoints, java.lang.Integer heightIndex, java.util.ArrayList<java.lang.Integer> outlierList) Computes the 2D error between the actual image points and the modeled image points
static Mat	computeVirtualCameraMatrix(Mat physicalCameraMatrix, Mat distortionCoefficients, Mat rectification, Size phyCamSize, Size virCamSize, double alpha, boolean keepPrincipalPoint) Computes the virtual camera matrix.
static Mat	computeVirtualCameraMatrix(Mat physicalCameraMatrix, Mat distortionCoefficients, Mat rectification, Size phyCamSize, Size virCamSize, double alpha, Mat principalPoint) Computes the virtual camera matrix.
static Mat	generateErrorImage(Size size, int testPatternIndex, double[][][] actual2DPoints, double[][][] modeled2DPoints, java.util.ArrayList<java.lang.Integer> outlierList) Generates a heat map of the residual errors between the actual image points and the modeled image points.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FIX_ASPECT_RATIO

public static final int FIX_ASPECT_RATIO

See Also:

Constant Field Values

FIX_PRINCIPAL_POINT

public static final int FIX_PRINCIPAL_POINT

See Also:

Constant Field Values

FIX_RADIAL_DISTORTION_COEFFICENTS

public static final int FIX_RADIAL_DISTORTION_COEFFICENTS

See Also:

Constant Field Values

FIX_TILT

public static final int FIX_TILT

See Also:

Constant Field Values

FIX_TANGENTIAL_DISTORTION_COEFFICENTS

public static final int FIX_TANGENTIAL_DISTORTION_COEFFICENTS

See Also:

Constant Field Values

sigmaThresholdForRejectingOutliers

public static final double sigmaThresholdForRejectingOutliers

See Also:

Constant Field Values

Constructor Detail

CameraCalibrationUtils

public CameraCalibrationUtils()

Method Detail

computeBestCameraParameters

public static double computeBestCameraParameters(double[][][] testPattern3dPoints,
                                                 double[][][] testPatternImagePoints,
                                                 double[][][] modeledImagePoints,
                                                 java.util.List<java.lang.Integer> outlierPointList,
                                                 double[] parameters)
                                          throws java.lang.Exception

Computes the best, in a least squared error sense, set of camera parameters that transform the given test pattern 3D coordinates to the corresponding 2D image points. This is very similar to what OpenCv's Calib3d.calibrateCamera does except this implementation takes advantage of the fact that it is known that the rotation of the camera WRT the test patterns is constant and that the Z height of the camera is also constant for all test patterns. The openpnp camera model that transforms 3D machine coordinates to 2D image points is: U = K * D( R * (X - T) ) Where: U is a 3x1 vector containing the image coordinates, in pixels, of a point (Z = 1 for all points); K is the 3x3 intrinsic camera matrix; D() is OpenCV's five parameter distortion model - takes a 3x1 vector as a parameter in undistorted camera coordinates and returns a normalized 3x1 vector (Z=1) in distorted camera coordinates; R is a 3x3 rotation matrix that transforms vectors from the machine reference frame to the camera reference frame; X is a 3x1 vector containing the machine coordinates of the same point as in U; and T is a 3x1 vector containing the machine coordinates of the camera Note that this is very similar as to how OpenCV models a camera except that here T is in the machine reference frame rather than in the camera's reference frame. This is so that we can treat the Z component of T as a single constant (since we know the camera is not changing in Z during the calibration process). The objective of the calibration process to use a set of known U's and X's to solve for the unknown K, D(), R, and T (see the description of parameters below)

Parameters:

testPattern3dPoints - - a numberOfTestPatterns x numberOfPointsPerTestPattern x 3 array containing the 3D machine coordinates at which the corresponding point in testPatternImagePoints was collected.

testPatternImagePoints - - a numberOfTestPatterns x numberOfPointsPerTestPattern x 2 array containing the 2D image coordinates of the corresponding point in testPattern3dPoints.

modeledImagePoints - - a numberOfTestPatterns x numberOfPointsPerTestPattern x 2 array that, upon return of this method, will contain the 2D modeled image coordinates of the corresponding points in testPattern3dPoints.

outlierPointList - - an output list of indices to points that were excluded from the parameter estimation due to being possible outliers.

parameters - - a 13+2*numberOfTestPatterns element array containing the camera parameters to be estimated in the following order: fx, fy, cx, cy, k1, k2, p1, p2, k3, Rx, Ry, Rz, cam_z, cam_x[0], cam_y[0], ... cam_x[numberOfTestPatterns-1], cam_y[numberOfTestPatterns-1]. Where fx, fy, cx, and cy are the intrinsic camera matrix components; k1, k2, p1, p2, and k3 are the intrinsic camera lens distortion coefficients; Rx, Ry, and Rz are the extrinsic camera rotation vector components; cam_z is the camera Z machine coordinate; and cam_x[0], cam_y[0], ... cam_x[numberOfTestPatterns-1], and cam_y[numberOfTestPatterns-1] are the camera X/Y machine coordinates for each test pattern. This array needs to be initialized with approximately correct values prior to calling this method and upon return of this method will contain the best fit camera parameters.

Returns:

the DRMS error in pixels of the model's fit to the data points.

Throws:

java.lang.Exception

computeBestCameraParameters

public static double computeBestCameraParameters(double[][][] testPattern3dPoints,
                                                 double[][][] testPatternImagePoints,
                                                 double[][][] modeledImagePoints,
                                                 java.util.List<java.lang.Integer> outlierPointList,
                                                 double[] parameters,
                                                 int flags)
                                          throws java.lang.Exception

Computes the best, in a least squared error sense, set of camera parameters that transform the given test pattern 3D coordinates to the corresponding 2D image points. This is very similar to what OpenCv's Calib3d.calibrateCamera does except this implementation takes advantage of the fact that it is known that the rotation of the camera WRT the test patterns is constant and that the Z height of the camera is also constant for all test patterns. The openpnp camera model that transforms 3D machine coordinates to 2D image points is: U = K * D( R * (X - T) ) Where: U is a 3x1 vector containing the image coordinates, in pixels, of a point (Z = 1 for all points); K is the 3x3 intrinsic camera matrix; D() is OpenCV's five parameter distortion model - takes a 3x1 vector as a parameter in undistorted camera coordinates and returns a normalized 3x1 vector (Z=1) in distorted camera coordinates; R is a 3x3 rotation matrix that transforms vectors from the machine reference frame to the camera reference frame; X is a 3x1 vector containing the machine coordinates of the same point as in U; and T is a 3x1 vector containing the machine coordinates of the camera Note that this is very similar as to how OpenCV models a camera except that here T is in the machine reference frame rather than in the camera's reference frame. This is so that we can treat the Z component of T as a single constant (since we know the camera is not changing in Z during the calibration process). The objective of the calibration process to use a set of known U's and X's to solve for the unknown K, D(), R, and T (see the description of parameters below)

Parameters:

testPattern3dPoints - - a numberOfTestPatterns x numberOfPointsPerTestPattern x 3 array containing the 3D machine coordinates at which the corresponding point in testPatternImagePoints was collected.

testPatternImagePoints - - a numberOfTestPatterns x numberOfPointsPerTestPattern x 2 array containing the 2D image coordinates of the corresponding point in testPattern3dPoints.

modeledImagePoints - - a numberOfTestPatterns x numberOfPointsPerTestPattern x 2 array that, upon return of this method, will contain the 2D modeled image coordinates of the corresponding points in testPattern3dPoints.

outlierPointList - - an output list of indices to points that were excluded from the parameter estimation due to being possible outliers.

parameters - - a 13+2*numberOfTestPatterns element array containing the camera parameters to be estimated in the following order: fx, fy, cx, cy, k1, k2, p1, p2, k3, Rx, Ry, Rz, cam_z, cam_x[0], cam_y[0], ... cam_x[numberOfTestPatterns-1], cam_y[numberOfTestPatterns-1]. Where fx, fy, cx, and cy are the intrinsic camera matrix components; k1, k2, p1, p2, and k3 are the intrinsic camera lens distortion coefficients; Rx, Ry, and Rz are the extrinsic camera rotation vector components; cam_z is the camera Z machine coordinate; and cam_x[0], cam_y[0], ... cam_x[numberOfTestPatterns-1], and cam_y[numberOfTestPatterns-1] are the camera X/Y machine coordinates for each test pattern. This array needs to be initialized with approximately correct values prior to calling this method and upon return of this method will contain the best fit camera parameters.

flags - - Flags used to force certain parameters be retained from the initial starting values. Can be one or more of the follow added/or'ed together: FIX_ASPECT_RATIO, FIX_CENTER_POINT, FIX_RADIAL_DISTORTION_COEFFICENTS, FIX_TILT, FIX_TANGENTIAL_DISTORTION_COEFFICENTS.

Returns:

the DRMS error in pixels of the model's fit to the data points.

Throws:

java.lang.Exception

computeRectificationMatrix

public static Mat computeRectificationMatrix(Mat transformFromMachToPhyCamRefFrame,
                                             Mat vectorFromMachToPhyCamInMachRefFrame,
                                             Mat transformFromMachToVirCamRefFrame,
                                             Mat vectorFromMachToVirCamInMachRefFrame,
                                             double primaryZ)

Computes the rectification matrix that converts normalized physical camera coordinates to the virtual camera coordinates

Parameters:

transformFromMachToPhyCamRefFrame - - the 3x3 rotation matrix that converts vector components from the machine coordinate system to the physical camera coordinate system

vectorFromMachToPhyCamInMachRefFrame - - the 3x1 vector from the machine origin to the physical camera origin with components represented in the machine coordinate system

transformFromMachToVirCamRefFrame - - the 3x3 rotation matrix that converts vector components from the machine coordinate system to the virtual camera coordinate system

vectorFromMachToVirCamInMachRefFrame - - the 3x1 vector from the machine origin to the virtual camera origin with components represented in the machine coordinate system

primaryZ - - the machine Z coordinate, in millimeters, that is used as the default imaging height

Returns:

computeVirtualCameraMatrix

public static Mat computeVirtualCameraMatrix(Mat physicalCameraMatrix,
                                             Mat distortionCoefficients,
                                             Mat rectification,
                                             Size phyCamSize,
                                             Size virCamSize,
                                             double alpha,
                                             boolean keepPrincipalPoint)

Computes the virtual camera matrix. This is similar to openCV's getOptimalNewCameraMatrix except that it also takes into account any image rectification.

Parameters:

physicalCameraMatrix - - the physical camera's intrinsic matrix

distortionCoefficients - - the physical camera's lens distortion coefficients

rectification - - the rectification matrix that takes the physical camera points to the virtual camera points

size - - the physical camera's image size

alpha - - a free scaling parameter in the range 0 to 1 inclusive. A value of a zero ensures only valid image pixels are displayed but may result in the loss of some valid pixels around the edge of the image. A value of one ensure all valid pixels are displayed but that may result in some invalid (usually black) pixels being displayed around the edge of the image.

keepPrincipalPoint - - if set true, the virtual camera's principal point is set to match that of the physical camera's principal point

Returns:

the virtual camera's intrinsic camera matrix

computeVirtualCameraMatrix

public static Mat computeVirtualCameraMatrix(Mat physicalCameraMatrix,
                                             Mat distortionCoefficients,
                                             Mat rectification,
                                             Size phyCamSize,
                                             Size virCamSize,
                                             double alpha,
                                             Mat principalPoint)

Computes the virtual camera matrix. This is similar to openCV's getOptimalNewCameraMatrix except that it also takes into account image rectification.

Parameters:

physicalCameraMatrix - - the physical camera's intrinsic matrix

distortionCoefficients - - the physical camera's lens distortion coefficients

rectification - - the rectification matrix that takes the physical camera points to the virtual camera points

phyCamSize - - the physical camera's image size

virCamSize - - (output) the virtual camera's image size

alpha - - a free scaling parameter in the range 0 to 1 inclusive. A value of a zero ensures only valid image pixels are displayed but may result in the loss of some valid pixels around the edge of the image. A value of one ensure all valid pixels are displayed but that may result in some invalid (usually black) pixels being displayed around the edge of the image.

principalPoint - - a 3x1 matrix containing the point in physical camera coordinates of the desired principal point of the virtual camera, if null, the principal point will be computed so as to maximize the valid pixels in the resulting image

Returns:

the virtual camera's intrinsic camera matrix

computeResidualErrors

public static java.util.List<double[]> computeResidualErrors(double[][][] actual2DPoints,
                                                             double[][][] modeled2DPoints)

Computes the 2D error between the actual image points and the modeled image points

Parameters:

actual2DPoints - - the actual image points

modeled2DPoints - - the modeled image points

Returns:

the 2D residual errors

computeResidualErrors

public static java.util.List<double[]> computeResidualErrors(double[][][] actual2DPoints,
                                                             double[][][] modeled2DPoints,
                                                             java.util.ArrayList<java.lang.Integer> outlierList)

Computes the 2D error between the actual image points and the modeled image points

Parameters:

actual2DPoints - - the actual image points

modeled2DPoints - - the modeled image points

outlierList - - a list of outlier points to exclude, set to null to not exclude any points

Returns:

the 2D residual errors

computeResidualErrors

public static java.util.List<double[]> computeResidualErrors(double[][][] actual2DPoints,
                                                             double[][][] modeled2DPoints,
                                                             java.lang.Integer heightIndex)

Computes the 2D error between the actual image points and the modeled image points

Parameters:

actual2DPoints - - the actual image points

modeled2DPoints - - the modeled image points

heightIndex - - index to select which calibration height to compute the residual errors, set to null to compute over all heights

Returns:

the 2D residual errors

computeResidualErrors

public static java.util.List<double[]> computeResidualErrors(double[][][] actual2DPoints,
                                                             double[][][] modeled2DPoints,
                                                             java.lang.Integer heightIndex,
                                                             java.util.ArrayList<java.lang.Integer> outlierList)

Computes the 2D error between the actual image points and the modeled image points

Parameters:

actual2DPoints - - the actual image points

modeled2DPoints - - the modeled image points

heightIndex - - index to select which calibration height to compute the residual errors, set to null to compute over all heights

outlierList - - a list of outlier points to exclude, set to null to not exclude any points

Returns:

the 2D residual errors

computeDrmsError

public static double computeDrmsError(java.util.List<double[]> residuals)

Computes the Distance Root-Mean-Squared (DRMS) error for a given set of 2D errors

Parameters:

residuals - - the errors

Returns:

the DRMS value of the errors

generateErrorImage

public static Mat generateErrorImage(Size size,
                                     int testPatternIndex,
                                     double[][][] actual2DPoints,
                                     double[][][] modeled2DPoints,
                                     java.util.ArrayList<java.lang.Integer> outlierList)

Generates a heat map of the residual errors between the actual image points and the modeled image points. The image is scaled so that darkest red indicates the areas of highest error and darkest blue zero error.

Parameters:

size - - the image size

testPatternIndex - - selects which calibration height for which the image is generated

actual2DPoints - - the actual 2D image points

modeled2DPoints - - the modeled 2D image points

outlierList - - a list of outlier points to exclude from the image, set to null to not exclude any points

Returns:

the heat map of errors