本次课程内容 相机成像模型 小孔成像、相机成像参数 单目视觉定位（测距）方法 局限性、难点 双目视觉方法 原理

“午”即小孔所在处。这段文字表明小孔成的是倒像，其原因是在小孔处光线交叉的地方有一点（“端”），成像的大小，与这交点的位置无关。

一般情况下单相机无法正确获得深度信息！

特殊情况下单相机可以计算深度信息 比如：x 或 y 已知

简单计算球位置的方法！ （x,y） Z Y （X,Y,Z） 如果空间点的 Y 坐标已知,应用 可以计算出空间点的距离 Z Image plane f Y （X,Y,Z） 如果空间点的 Y 坐标已知,应用 可以计算出空间点的距离 Z 简单计算球位置的方法！

简单计算位置的方法 （x,y） Z Y （X,Y,Z） Y取机器人身高（或实际测量相机高度），y可以从 图像读取，f 可以标定，则 ，同样也

单目方法存在的问题 （x,y） Z Y （X,Y,Z） Y（即相机距离地面高度）无法精确测量 相机主光轴与地面夹角无法测量 f Y （X,Y,Z） Y（即相机距离地面高度）无法精确测量 相机主光轴与地面夹角无法测量 导致距离 Z 不准确。

像机标定

坐标系 y u x v O 1、世界坐标系： 2、摄像机坐标系： 3、图像坐标系: 世界坐标系 说明： 为了校正成像畸变 用理想图像坐标系 和真实图像坐标系 分别描述畸变前后的坐标关系

摄像机光学成像过程的四个步骤 世界坐标系 刚体变换 摄像机坐标系 透视投影 理想图像坐标系 畸变校正 真实图像坐标系 数字化图像 1、刚体变换公式 刚体变换 透视投影 畸变校正 数字化图像 世界坐标系 摄像机坐标系 真实图像坐标系 数字化图像坐标系 理想图像坐标系 齐次坐标形式

2、透视投影——透镜成像原理图 一般地由于 于是 这时可以将透镜成像模型近似地用小孔模型代替 物体 O C B A 图像 一般地由于 于是 这时可以将透镜成像模型近似地用小孔模型代替 f=OB 为透镜的焦距 m=OC 为像距 n=AO 为物距

2、透视投影——小孔成像模型 o 写成齐次坐标形式为

2、中心透视投影模型 o f 写成齐次坐标形式为

3、畸变校正——径向和切向畸变 径向畸变 径向失真 离心畸变 切向失真 薄透镜畸变 dr :radial distortion Ideal Position Position with distortion dr :radial distortion dt :tangential distortion

3、畸变校正——其它畸变类型 现在常用模型 a b a :barrel distortion b :pincushion distortion a b 桶形畸变a和枕形畸变b 桶形畸变 枕形畸变 现在常用模型

3、畸变校正——其它畸变类型

4、图像数字化 在 中的坐标为 象素在轴上的物理尺寸为 C V Affine Transformation : 齐次坐标形式: U 其中

摄像机的内参数矩阵 K

线性摄像机成像模型 图像像素坐标系 图像物理坐标系 摄像机坐标系 世界坐标系 最终得到： 图像像素坐标系 世界坐标系 这是忽略畸变的线性成像模型

张正友的平面标定方法 http://research.microsoft.com/en-us/um/people/zhang/Calib/

张正友的平面标定方法 基本原理： 在这里假定模板平面在世界坐标系 的平面上 在这里假定模板平面在世界坐标系 的平面上 其中， 为摄像机的内参数矩阵， 为模板平面上点的齐次坐标， 为模板平面上点投影到图象平面上对应点的齐次坐标， 和 分别是摄像机坐标系相对于世界坐标系的旋转矩阵和平移向量

张正友的平面标定方法 其中 根据旋转矩阵的性质，即 和 ，每幅图象可以获得以下两个对内参数矩阵的基本约束 根据旋转矩阵的性质，即 和 ，每幅图象可以获得以下两个对内参数矩阵的基本约束 由于摄像机有5个未知内参数，所以当所摄取得的图象数目大于等于3时，就可以线性唯一求解出

张正友的平面标定方法 张正友方法所用的平面模板

张正友的平面标定方法 算法描述 打印一张模板并贴在一个平面上 从不同角度拍摄若干张模板图象 检测出图象中的特征点 求出摄像机的内参数和外参数 求出畸变系数 优化求精

http://www. vision. caltech. edu/bouguetj/calib_doc/htmls/example2 http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example2.html

其它标定方法： http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/links.html

多视立体视觉(测距)方法

Review: Pinhole Camera

Review: Perspective Projection

Review: Intrinsic Camera Parameters Y M Image plane Z C v X m Focal plane u

Review: Extrinsic Parameters Y M Image plane Y Z C v X X Z m Focal plane u By Rigid Body Transformation:

Review: Perspective Projection Points go to Points Lines go to Lines Planes go to whole image or Half-planes Polygons go to Polygons 但是, 平行线相交!

Perspective cues

Perspective cues

Perspective cues

Recovering 3D from images What cues in the image provide 3D information?

Merle Norman Cosmetics, Los Angeles Visual cues Shading Merle Norman Cosmetics, Los Angeles

The Visual Cliff, by William Vandivert, 1960 Visual cues Shading Texture The Visual Cliff, by William Vandivert, 1960

From The Art of Photography, Canon Visual cues Shading Texture Focus From The Art of Photography, Canon

Visual cues Shading Texture Focus Motion

Visual cues Shading Texture Focus Motion Shape From X X = shading, texture, focus, motion, ...

Fundamentals of Stereo Vision A camera model: Models how 3-D scene points are transformed into 2-D image points The pinhole camera: a simple linear model for perspective projection

Fundamentals of Stereo Vision The goal of stereo analysis: The inverse process: From 2-D image coordinates to 3-D scene coordinates Requires images from at least two views

Fundamentals of Stereo Vision 3-D reconstruction

Fundamentals of Stereo Vision

Fundamentals of Stereo Vision

Fundamentals of Stereo Vision

Fundamentals of Stereo Vision

Fundamentals of Stereo Vision

Fundamentals of Stereo Vision

Fundamentals of Stereo Vision

Multi-View Geometry Relates 3D World Points Camera Centers Camera Orientations Camera Centers

Multi-View Geometry Relates 3D World Points Camera Centers Camera Intrinsic Parameters Image Points Camera Orientations

Binocular Stereo Gives reconstruction as intersection of two rays scene point image plane optical center Basic Principle: Triangulation Gives reconstruction as intersection of two rays Requires calibration point correspondence

Stereo_Two subproblems Matching (hardest) Finding corresponding elements in the two images Reconstruction Establishing 3-D coordinates from the 2-D image correspondences found during matching

Stereo Constraints p’ p ? Given p in left image, where can the corresponding point p’ in right image be?

Stereo Constraints M Image plane Epipolar Line Y1 p p’ Y2 X2 O1 Z1 X1 Epipole Focal plane

Epipolar Constraint

From Geometry to Algebra P p p’

Reconstruction up to a Scale Factor Assume that intrinsic parameters of both cameras are known Essential Matrix is known up to a scale factor (for example, estimated from the 8 point algorithm).

From Geometry to Algebra P p p’

Linear Constraint: Should be able to express as matrix multiplication.

Review: Matrix Form of Cross Product 正交

Review: Matrix Form of Cross Product

Matrix Form

The Essential Matrix

Reconstruction O O’ p p’

Reconstruction Equation 1 Equation 2 (From equations 1 and 2)

Stereo image rectification Image Reprojection reproject image planes onto common plane parallel to line between optical centers a homography (3x3 transform) applied to both input images pixel motion is horizontal after this transformation C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.

Image Rectification Common Image Plane Parallel Epipolar Lines Search Correspondences on scan line

A Simple Stereo System Right image: Left image: target reference LEFT CAMERA RIGHT CAMERA baseline Elevation Zw disparity Depth Z Right image: target Left image: reference Zw=0 Bahadir K. Gunturk

Stereo View Left View Right View Disparity

Stereo Disparity The separation between two matching objects is called the stereo disparity.

Parallel Cameras P Z xl xr f pl pr Ol Or Disparity: T T is the stereo baseline

Correlation Approach LEFT IMAGE (xl, yl) (0). Essential Equation represents actually the epipolar plane in either the left or the right image (1). Epipolar line in the right image given pl (Epl)Tpr=0 zr = fr extension of the equations in pr = (xr,yr,fr) (2). Epipolar line in the left image given pr (prTE) pl=0 zl = fl For Each point (xl, yl) in the left image, define a window centered at the point Bahadir K. Gunturk

Correlation Approach RIGHT IMAGE (xl, yl) (0). Essential Equation represents actually the epipolar plane in either the left or the right image (1). Epipolar line in the right image given pl (Epl)Tpr=0 zr = fr extension of the equations in pr = (xr,yr,fr) (2). Epipolar line in the left image given pr (prTE) pl=0 zl = fl … search its corresponding point within a search region in the right image Bahadir K. Gunturk

Correlation Approach RIGHT IMAGE (xr, yr) dx (xl, yl) (0). Essential Equation represents actually the epipolar plane in either the left or the right image (1). Epipolar line in the right image given pl (Epl)Tpr=0 zr = fr extension of the equations in pr = (xr,yr,fr) (2). Epipolar line in the left image given pr (prTE) pl=0 zl = fl … the disparity (dx, dy) is the displacement when the correlation is maximum Bahadir K. Gunturk

Comparing Windows ? = g f Minimize Sum of Squared Differences Maximize Cross correlation

Correspondence using Discrete Search

Sum of Squared Differences (SSD)

Feature-based correspondence Features most commonly used: Corners Similarity measured in terms of: surrounding gray values (SSD, Cross-correlation) location Edges, Lines orientation contrast coordinates of edge or line’s midpoint length of line

Feature-based Approach LEFT IMAGE corner line structure (0). Essential Equation represents actually the epipolar plane in either the left or the right image (1). Epipolar line in the right image given pl (Epl)Tpr=0 zr = fr extension of the equations in pr = (xr,yr,fr) (2). Epipolar line in the left image given pr (prTE) pl=0 zl = fl For each feature in the left image…

Feature-based Approach RIGHT IMAGE corner line structure (0). Essential Equation represents actually the epipolar plane in either the left or the right image (1). Epipolar line in the right image given pl (Epl)Tpr=0 zr = fr extension of the equations in pr = (xr,yr,fr) (2). Epipolar line in the left image given pr (prTE) pl=0 zl = fl Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

Correspondence Difficulties Why is the correspondence problem difficult? Some points in each image will have no corresponding points in the other image. (1) the cameras might have different fields of view. (2) due to occlusion. A stereo system must be able to determine the image parts that should not be matched.

Stereo results Data from University of Tsukuba Scene Ground truth (Seitz)

Results with window correlation Estimated depth of field Ground truth (Seitz)

Results with better method A state of the art method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999. Ground truth (Seitz)

Other constraints It is possible to put some constraints. For example: smoothness. (Disparity usually doesn’t change too quickly.)

Reconstruction up to a Scale Factor Assume that intrinsic parameters of both cameras are known Essential Matrix is known up to a scale factor (for example, estimated from the 8 point algorithm).

Reconstruction up to a Scale Factor

Reconstruction up to a Scale Factor Let It can be proved that

Reconstruction up to a Scale Factor We have two choices of t, (t+ and t-) because of sign ambiguity and two choices of E, (E+ and E-). This gives us four pairs of translation vectors and rotation matrices.

Reconstruction up to a Scale Factor Given and Construct the vectors w, and compute R Reconstruct the Z and Z’ for each point If the signs of Z and Z’ of the reconstructed points are both negative for some point, change the sign of and go to step 2. different for some point, change the sign of each entry of and go to step 1. both positive for all points, exit.