Difference between revisions of "Example Rectification Calibrated"

From BoofCV
Jump to navigationJump to search
m
m
Line 8: Line 8:
Stereo rectification is the process of distorting an image such that the epipoles of both images are at infinity. If the epipoles are at infinity along the x-axis, then corresponding features must lie along the same y-coordinates. Using knowledge that correspondence feature's have the same y-coordinate allows for quick searches. Many stereo vision algorithm rely on rectification.  The example below demonstrates rectification for a calibrated stereo pair. Note that after calibration the new camera view has a different set of intrinsic parameters.
Stereo rectification is the process of distorting an image such that the epipoles of both images are at infinity. If the epipoles are at infinity along the x-axis, then corresponding features must lie along the same y-coordinates. Using knowledge that correspondence feature's have the same y-coordinate allows for quick searches. Many stereo vision algorithm rely on rectification.  The example below demonstrates rectification for a calibrated stereo pair. Note that after calibration the new camera view has a different set of intrinsic parameters.


Example File: [https://github.com/lessthanoptimal/BoofCV/blob/v0.36/examples/src/main/java/boofcv/examples/stereo/ExampleRectifyCalibratedStereo.java ExampleRectifyCalibratedStereo.java]
Example File: [https://github.com/lessthanoptimal/BoofCV/blob/v0.37/examples/src/main/java/boofcv/examples/stereo/ExampleRectifyCalibratedStereo.java ExampleRectifyCalibratedStereo.java]


Concepts:
Concepts:
Line 21: Line 21:
<syntaxhighlight lang="java">
<syntaxhighlight lang="java">
/**
/**
* <p>
  * Shows how to rectify a pair of stereo images with known intrinsic parameters and stereo baseline. When you
  * Shows how to rectify a pair of stereo images with known intrinsic parameters and stereo baseline.
  * rectify a stereo pair you are applying a transform that removes lens distortion and "rotates" the views
  * The example code does the following:<br>
  * such that they are parallel to each other, facilitating stereo processing.
* 1) Load stereo parameters from XML file with a pair of images.<br>
* 2) Undistort and rectify images..  This provides one rectification matrix
  * for each image along with a new camera calibration matrix.<br>
* 3) The original rectification does not try to maximize view area, however it can be adjusted.
* 4)After rectification is finished the results are displayed.<br>
* </p>
  *
  *
  * <p>
  * The example code does the following:
  * Note that the y-axis in left and right images align after rectification.  The curved image edge
* <lo>
* is an artifact of lens distortion being removed.
*    <li>Load stereo extrinsic and intrinsic parameters from a file along with a pair of images.</li>
* </p>
*    <li>Undistort and rectify images. This provides one rectification matrix for each image along with a new
*        camera calibration matrix.</li>
*    <li>The original rectification does not try to maximize view area, however it can be adjusted.</li>
*    <li>After rectification is finished the results are displayed.</li>
* </lo>
*
  * Note that the y-axis in left and right images align after rectification. You can click in the images to draw a line
  * that makes this easy to see. The curved image birder is an artifact of lens distortion being removed.
  *
  *
  * @author Peter Abeles
  * @author Peter Abeles
  */
  */
public class ExampleRectifyCalibratedStereo {
public class ExampleRectifyCalibratedStereo {
 
public static void main( String[] args ) {
public static void main( String args[] ) {
String dir = UtilIO.pathExample("calibration/stereo/Bumblebee2_Chess/");
String dir = UtilIO.pathExample("calibration/stereo/Bumblebee2_Chess/");


StereoParameters param = CalibrationIO.load(new File(dir , "stereo.yaml"));
StereoParameters param = CalibrationIO.load(new File(dir, "stereo.yaml"));


// load images
// load images
BufferedImage origLeft = UtilImageIO.loadImage(dir,"left05.jpg");
BufferedImage origLeft = UtilImageIO.loadImage(dir, "left05.jpg");
BufferedImage origRight = UtilImageIO.loadImage(dir,"right05.jpg");
BufferedImage origRight = UtilImageIO.loadImage(dir, "right05.jpg");


// distorted images
// distorted images
Planar<GrayF32> distLeft =
Planar<GrayF32> distLeft =
ConvertBufferedImage.convertFromPlanar(origLeft, null,true, GrayF32.class);
ConvertBufferedImage.convertFromPlanar(origLeft, null, true, GrayF32.class);
Planar<GrayF32> distRight =
Planar<GrayF32> distRight =
ConvertBufferedImage.convertFromPlanar(origRight, null,true, GrayF32.class);
ConvertBufferedImage.convertFromPlanar(origRight, null, true, GrayF32.class);


// storage for undistorted + rectified images
// storage for undistorted + rectified images
Line 67: Line 67:
DMatrixRMaj K2 = PerspectiveOps.pinholeToMatrix(param.getRight(), (DMatrixRMaj)null);
DMatrixRMaj K2 = PerspectiveOps.pinholeToMatrix(param.getRight(), (DMatrixRMaj)null);


rectifyAlg.process(K1,new Se3_F64(),K2,leftToRight);
rectifyAlg.process(K1, new Se3_F64(), K2, leftToRight);


// rectification matrix for each image
// rectification matrix for each image
DMatrixRMaj rect1 = rectifyAlg.getRect1();
DMatrixRMaj rect1 = rectifyAlg.getUndistToRectPixels1();
DMatrixRMaj rect2 = rectifyAlg.getRect2();
DMatrixRMaj rect2 = rectifyAlg.getUndistToRectPixels2();
// New calibration matrix,
// New calibration matrix,
// Both cameras have the same one after rectification.
// Both cameras have the same one after rectification.
Line 81: Line 81:


// undistorted and rectify images
// undistorted and rectify images
FMatrixRMaj rect1_F32 = new FMatrixRMaj(3,3); // TODO simplify code some how
FMatrixRMaj rect1_F32 = new FMatrixRMaj(3, 3); // TODO simplify code some how
FMatrixRMaj rect2_F32 = new FMatrixRMaj(3,3);
FMatrixRMaj rect2_F32 = new FMatrixRMaj(3, 3);
ConvertMatrixData.convert(rect1, rect1_F32);
ConvertMatrixData.convert(rect1, rect1_F32);
ConvertMatrixData.convert(rect2, rect2_F32);
ConvertMatrixData.convert(rect2, rect2_F32);


ImageDistort rectifyImageLeft =
ImageDistort rectifyImageLeft =
RectifyImageOps.rectifyImage(param.getLeft(), rect1_F32, BorderType.SKIP, distLeft.getImageType());
RectifyDistortImageOps.rectifyImage(param.getLeft(), rect1_F32, BorderType.SKIP, distLeft.getImageType());
ImageDistort rectifyImageRight =
ImageDistort rectifyImageRight =
RectifyImageOps.rectifyImage(param.getRight(), rect2_F32, BorderType.SKIP, distRight.getImageType());
RectifyDistortImageOps.rectifyImage(param.getRight(), rect2_F32, BorderType.SKIP, distRight.getImageType());


rectifyImageLeft.apply(distLeft,rectLeft);
rectifyImageLeft.apply(distLeft, rectLeft);
rectifyImageRight.apply(distRight,rectRight);
rectifyImageRight.apply(distRight, rectRight);


// convert for output
// convert for output
BufferedImage outLeft = ConvertBufferedImage.convertTo(rectLeft,null,true);
BufferedImage outLeft = ConvertBufferedImage.convertTo(rectLeft, null, true);
BufferedImage outRight = ConvertBufferedImage.convertTo(rectRight, null,true);
BufferedImage outRight = ConvertBufferedImage.convertTo(rectRight, null, true);


// show results and draw a horizontal line where the user clicks to see rectification easier
// show results and draw a horizontal line where the user clicks to see rectification easier
Line 103: Line 103:
panel.addItem(new RectifiedPairPanel(true, outLeft, outRight), "Rectified");
panel.addItem(new RectifiedPairPanel(true, outLeft, outRight), "Rectified");


ShowImages.showWindow(panel,"Stereo Rectification Calibrated",true);
ShowImages.showWindow(panel, "Stereo Rectification Calibrated", true);
}
}
}
}
</syntaxhighlight>
</syntaxhighlight>

Revision as of 19:48, 21 December 2020

Stereo rectification is the process of distorting an image such that the epipoles of both images are at infinity. If the epipoles are at infinity along the x-axis, then corresponding features must lie along the same y-coordinates. Using knowledge that correspondence feature's have the same y-coordinate allows for quick searches. Many stereo vision algorithm rely on rectification. The example below demonstrates rectification for a calibrated stereo pair. Note that after calibration the new camera view has a different set of intrinsic parameters.

Example File: ExampleRectifyCalibratedStereo.java

Concepts:

  • Stereo Rectification
  • Stereo Vision

Related Examples:

Example Code

/**
 * Shows how to rectify a pair of stereo images with known intrinsic parameters and stereo baseline. When you
 * rectify a stereo pair you are applying a transform that removes lens distortion and "rotates" the views
 * such that they are parallel to each other, facilitating stereo processing.
 *
 * The example code does the following:
 * <lo>
 *     <li>Load stereo extrinsic and intrinsic parameters from a file along with a pair of images.</li>
 *     <li>Undistort and rectify images. This provides one rectification matrix for each image along with a new
 *         camera calibration matrix.</li>
 *     <li>The original rectification does not try to maximize view area, however it can be adjusted.</li>
 *     <li>After rectification is finished the results are displayed.</li>
 * </lo>
 *
 * Note that the y-axis in left and right images align after rectification. You can click in the images to draw a line
 * that makes this easy to see. The curved image birder is an artifact of lens distortion being removed.
 *
 * @author Peter Abeles
 */
public class ExampleRectifyCalibratedStereo {
	public static void main( String[] args ) {
		String dir = UtilIO.pathExample("calibration/stereo/Bumblebee2_Chess/");

		StereoParameters param = CalibrationIO.load(new File(dir, "stereo.yaml"));

		// load images
		BufferedImage origLeft = UtilImageIO.loadImage(dir, "left05.jpg");
		BufferedImage origRight = UtilImageIO.loadImage(dir, "right05.jpg");

		// distorted images
		Planar<GrayF32> distLeft =
				ConvertBufferedImage.convertFromPlanar(origLeft, null, true, GrayF32.class);
		Planar<GrayF32> distRight =
				ConvertBufferedImage.convertFromPlanar(origRight, null, true, GrayF32.class);

		// storage for undistorted + rectified images
		Planar<GrayF32> rectLeft = distLeft.createSameShape();
		Planar<GrayF32> rectRight = distRight.createSameShape();

		// Compute rectification
		RectifyCalibrated rectifyAlg = RectifyImageOps.createCalibrated();
		Se3_F64 leftToRight = param.getRightToLeft().invert(null);

		// original camera calibration matrices
		DMatrixRMaj K1 = PerspectiveOps.pinholeToMatrix(param.getLeft(), (DMatrixRMaj)null);
		DMatrixRMaj K2 = PerspectiveOps.pinholeToMatrix(param.getRight(), (DMatrixRMaj)null);

		rectifyAlg.process(K1, new Se3_F64(), K2, leftToRight);

		// rectification matrix for each image
		DMatrixRMaj rect1 = rectifyAlg.getUndistToRectPixels1();
		DMatrixRMaj rect2 = rectifyAlg.getUndistToRectPixels2();
		// New calibration matrix,
		// Both cameras have the same one after rectification.
		DMatrixRMaj rectK = rectifyAlg.getCalibrationMatrix();

		// Adjust the rectification to make the view area more useful
		RectifyImageOps.fullViewLeft(param.left, rect1, rect2, rectK, null);
//		RectifyImageOps.allInsideLeft(param.left, rect1, rect2, rectK, null);

		// undistorted and rectify images
		FMatrixRMaj rect1_F32 = new FMatrixRMaj(3, 3); // TODO simplify code some how
		FMatrixRMaj rect2_F32 = new FMatrixRMaj(3, 3);
		ConvertMatrixData.convert(rect1, rect1_F32);
		ConvertMatrixData.convert(rect2, rect2_F32);

		ImageDistort rectifyImageLeft =
				RectifyDistortImageOps.rectifyImage(param.getLeft(), rect1_F32, BorderType.SKIP, distLeft.getImageType());
		ImageDistort rectifyImageRight =
				RectifyDistortImageOps.rectifyImage(param.getRight(), rect2_F32, BorderType.SKIP, distRight.getImageType());

		rectifyImageLeft.apply(distLeft, rectLeft);
		rectifyImageRight.apply(distRight, rectRight);

		// convert for output
		BufferedImage outLeft = ConvertBufferedImage.convertTo(rectLeft, null, true);
		BufferedImage outRight = ConvertBufferedImage.convertTo(rectRight, null, true);

		// show results and draw a horizontal line where the user clicks to see rectification easier
		ListDisplayPanel panel = new ListDisplayPanel();
		panel.addItem(new RectifiedPairPanel(true, origLeft, origRight), "Original");
		panel.addItem(new RectifiedPairPanel(true, outLeft, outRight), "Rectified");

		ShowImages.showWindow(panel, "Stereo Rectification Calibrated", true);
	}
}