本次课程内容 相机成像模型 小孔成像、相机成像参数 单目视觉定位（测距）方法 局限性、难点 双目视觉方法 原理.

Presentation on theme: "本次课程内容 相机成像模型 小孔成像、相机成像参数 单目视觉定位（测距）方法 局限性、难点 双目视觉方法 原理."— Presentation transcript:

1 本次课程内容 相机成像模型 小孔成像、相机成像参数 单目视觉定位（测距）方法 局限性、难点 双目视觉方法 原理

2 “午”即小孔所在处。这段文字表明小孔成的是倒像，其原因是在小孔处光线交叉的地方有一点（“端”），成像的大小，与这交点的位置无关。

3

4

5

6

7

8

9

10

11

12

13

14

15 一般情况下单相机无法正确获得深度信息！

16 特殊情况下单相机可以计算深度信息 比如：x 或 y 已知

17 简单计算球位置的方法！ （x,y） Z Y （X,Y,Z） 如果空间点的 Y 坐标已知,应用 可以计算出空间点的距离 Z Image
plane f Y （X,Y,Z） 如果空间点的 Y 坐标已知,应用 可以计算出空间点的距离 Z 简单计算球位置的方法！

18 简单计算位置的方法 （x,y） Z Y （X,Y,Z） Y取机器人身高（或实际测量相机高度），y可以从 图像读取，f 可以标定，则 ，同样也

19 单目方法存在的问题 （x,y） Z Y （X,Y,Z） Y（即相机距离地面高度）无法精确测量 相机主光轴与地面夹角无法测量
f Y （X,Y,Z） Y（即相机距离地面高度）无法精确测量 相机主光轴与地面夹角无法测量 导致距离 Z 不准确。

20 像机标定

21 坐标系 y u x v O 1、世界坐标系： 2、摄像机坐标系： 3、图像坐标系: 世界坐标系 说明： 为了校正成像畸变 用理想图像坐标系
和真实图像坐标系 分别描述畸变前后的坐标关系

22

23 摄像机光学成像过程的四个步骤 世界坐标系 刚体变换 摄像机坐标系 透视投影 理想图像坐标系 畸变校正 真实图像坐标系 数字化图像
1、刚体变换公式 刚体变换 透视投影 畸变校正 数字化图像 世界坐标系 摄像机坐标系 真实图像坐标系 数字化图像坐标系 理想图像坐标系 齐次坐标形式

24 2、透视投影——透镜成像原理图 一般地由于 于是 这时可以将透镜成像模型近似地用小孔模型代替 物体 O C B A 图像
一般地由于 于是 这时可以将透镜成像模型近似地用小孔模型代替 f=OB 为透镜的焦距 m=OC 为像距 n=AO 为物距

25 2、透视投影——小孔成像模型 o 写成齐次坐标形式为

26 2、中心透视投影模型 o f 写成齐次坐标形式为

27 3、畸变校正——径向和切向畸变 径向畸变 径向失真 离心畸变 切向失真 薄透镜畸变 dr :radial distortion
Ideal Position Position with distortion dr :radial distortion dt :tangential distortion

28 3、畸变校正——其它畸变类型 现在常用模型 a b a :barrel distortion
b :pincushion distortion a b 桶形畸变a和枕形畸变b 桶形畸变 枕形畸变 现在常用模型

29 3、畸变校正——其它畸变类型

30 4、图像数字化 在 中的坐标为 象素在轴上的物理尺寸为 C V Affine Transformation : 齐次坐标形式: U 其中

31 摄像机的内参数矩阵 K

32 线性摄像机成像模型 图像像素坐标系 图像物理坐标系 摄像机坐标系 世界坐标系 最终得到： 图像像素坐标系 世界坐标系
这是忽略畸变的线性成像模型

33 张正友的平面标定方法

34 张正友的平面标定方法 基本原理： 在这里假定模板平面在世界坐标系 的平面上
在这里假定模板平面在世界坐标系 的平面上 其中， 为摄像机的内参数矩阵， 为模板平面上点的齐次坐标， 为模板平面上点投影到图象平面上对应点的齐次坐标， 和 分别是摄像机坐标系相对于世界坐标系的旋转矩阵和平移向量

35 张正友的平面标定方法 其中 根据旋转矩阵的性质，即 和 ，每幅图象可以获得以下两个对内参数矩阵的基本约束
根据旋转矩阵的性质，即 和 ，每幅图象可以获得以下两个对内参数矩阵的基本约束 由于摄像机有5个未知内参数，所以当所摄取得的图象数目大于等于3时，就可以线性唯一求解出

36

37 张正友的平面标定方法 张正友方法所用的平面模板

38 张正友的平面标定方法 算法描述 打印一张模板并贴在一个平面上 从不同角度拍摄若干张模板图象 检测出图象中的特征点 求出摄像机的内参数和外参数
求出畸变系数 优化求精

39 http://www. vision. caltech. edu/bouguetj/calib_doc/htmls/example2

40

41

42

43

44

45

46

47

48 其它标定方法：

49 多视立体视觉(测距)方法

50 Review: Pinhole Camera

51 Review: Perspective Projection

52 Review: Intrinsic Camera Parameters
Y M Image plane Z C v X m Focal plane u

53 Review: Extrinsic Parameters
Y M Image plane Y Z C v X X Z m Focal plane u By Rigid Body Transformation:

54 Review: Perspective Projection
Points go to Points Lines go to Lines Planes go to whole image or Half-planes Polygons go to Polygons 但是, 平行线相交!

55 Perspective cues

56 Perspective cues

57 Perspective cues

58 Recovering 3D from images
What cues in the image provide 3D information?

59 Merle Norman Cosmetics, Los Angeles
Visual cues Shading Merle Norman Cosmetics, Los Angeles

60 The Visual Cliff, by William Vandivert, 1960
Visual cues Shading Texture The Visual Cliff, by William Vandivert, 1960

61 From The Art of Photography, Canon
Visual cues Shading Texture Focus From The Art of Photography, Canon

62 Visual cues Shading Texture Focus Motion

63 Visual cues Shading Texture Focus Motion
Shape From X X = shading, texture, focus, motion, ...

64 Fundamentals of Stereo Vision
A camera model: Models how 3-D scene points are transformed into 2-D image points The pinhole camera: a simple linear model for perspective projection

65 Fundamentals of Stereo Vision
The goal of stereo analysis: The inverse process: From 2-D image coordinates to 3-D scene coordinates Requires images from at least two views

66 Fundamentals of Stereo Vision
3-D reconstruction

67 Fundamentals of Stereo Vision

68 Fundamentals of Stereo Vision

69 Fundamentals of Stereo Vision

70 Fundamentals of Stereo Vision

71 Fundamentals of Stereo Vision

72 Fundamentals of Stereo Vision

73 Fundamentals of Stereo Vision

74 Multi-View Geometry Relates 3D World Points Camera Centers
Camera Orientations Camera Centers

75 Multi-View Geometry Relates 3D World Points Camera Centers
Camera Intrinsic Parameters Image Points Camera Orientations

76 Binocular Stereo Gives reconstruction as intersection of two rays
scene point image plane optical center Basic Principle: Triangulation Gives reconstruction as intersection of two rays Requires calibration point correspondence

77 Stereo_Two subproblems
Matching (hardest) Finding corresponding elements in the two images Reconstruction Establishing 3-D coordinates from the 2-D image correspondences found during matching

78 Stereo Constraints p’ p ?
Given p in left image, where can the corresponding point p’ in right image be?

79 Stereo Constraints M Image plane Epipolar Line Y1 p p’ Y2 X2 O1 Z1 X1
Epipole Focal plane

80 Epipolar Constraint

81 From Geometry to Algebra
P p p’

82 Reconstruction up to a Scale Factor
Assume that intrinsic parameters of both cameras are known Essential Matrix is known up to a scale factor (for example, estimated from the 8 point algorithm).

83 From Geometry to Algebra
P p p’

84 Linear Constraint: Should be able to express as matrix multiplication.

85 Review: Matrix Form of Cross Product
正交

86 Review: Matrix Form of Cross Product

87 Matrix Form

88 The Essential Matrix

89 Reconstruction O O’ p p’

90 Reconstruction Equation 1 Equation 2 (From equations 1 and 2)

91 Stereo image rectification
Image Reprojection reproject image planes onto common plane parallel to line between optical centers a homography (3x3 transform) applied to both input images pixel motion is horizontal after this transformation C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.

92 Image Rectification Common Image Plane Parallel Epipolar Lines
Search Correspondences on scan line

93 A Simple Stereo System Right image: Left image: target reference
LEFT CAMERA RIGHT CAMERA baseline Elevation Zw disparity Depth Z Right image: target Left image: reference Zw=0 Bahadir K. Gunturk

94 Stereo View Left View Right View Disparity

95 Stereo Disparity The separation between two matching objects is called the stereo disparity.

96 Parallel Cameras P Z xl xr f pl pr Ol Or Disparity: T
T is the stereo baseline

97 Correlation Approach LEFT IMAGE (xl, yl) (0). Essential Equation represents actually the epipolar plane in either the left or the right image (1). Epipolar line in the right image given pl (Epl)Tpr=0 zr = fr extension of the equations in pr = (xr,yr,fr) (2). Epipolar line in the left image given pr (prTE) pl=0 zl = fl For Each point (xl, yl) in the left image, define a window centered at the point Bahadir K. Gunturk

98 Correlation Approach RIGHT IMAGE (xl, yl) (0). Essential Equation represents actually the epipolar plane in either the left or the right image (1). Epipolar line in the right image given pl (Epl)Tpr=0 zr = fr extension of the equations in pr = (xr,yr,fr) (2). Epipolar line in the left image given pr (prTE) pl=0 zl = fl … search its corresponding point within a search region in the right image Bahadir K. Gunturk

99 Correlation Approach RIGHT IMAGE (xr, yr) dx (xl, yl) (0). Essential Equation represents actually the epipolar plane in either the left or the right image (1). Epipolar line in the right image given pl (Epl)Tpr=0 zr = fr extension of the equations in pr = (xr,yr,fr) (2). Epipolar line in the left image given pr (prTE) pl=0 zl = fl … the disparity (dx, dy) is the displacement when the correlation is maximum Bahadir K. Gunturk

100 Comparing Windows ? = g f Minimize Sum of Squared Differences Maximize
Cross correlation

101 Correspondence using Discrete Search

102 Sum of Squared Differences (SSD)

103 Feature-based correspondence
Features most commonly used: Corners Similarity measured in terms of: surrounding gray values (SSD, Cross-correlation) location Edges, Lines orientation contrast coordinates of edge or line’s midpoint length of line

104 Feature-based Approach
LEFT IMAGE corner line structure (0). Essential Equation represents actually the epipolar plane in either the left or the right image (1). Epipolar line in the right image given pl (Epl)Tpr=0 zr = fr extension of the equations in pr = (xr,yr,fr) (2). Epipolar line in the left image given pr (prTE) pl=0 zl = fl For each feature in the left image…

105 Feature-based Approach
RIGHT IMAGE corner line structure (0). Essential Equation represents actually the epipolar plane in either the left or the right image (1). Epipolar line in the right image given pl (Epl)Tpr=0 zr = fr extension of the equations in pr = (xr,yr,fr) (2). Epipolar line in the left image given pr (prTE) pl=0 zl = fl Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

106 Correspondence Difficulties
Why is the correspondence problem difficult? Some points in each image will have no corresponding points in the other image. (1) the cameras might have different fields of view. (2) due to occlusion. A stereo system must be able to determine the image parts that should not be matched.

107 Stereo results Data from University of Tsukuba Scene Ground truth
(Seitz)

108 Results with window correlation
Estimated depth of field Ground truth (Seitz)

109 Results with better method
A state of the art method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999. Ground truth (Seitz)

110 Other constraints It is possible to put some constraints.
For example: smoothness. (Disparity usually doesn’t change too quickly.)

111 Reconstruction up to a Scale Factor
Assume that intrinsic parameters of both cameras are known Essential Matrix is known up to a scale factor (for example, estimated from the 8 point algorithm).

112 Reconstruction up to a Scale Factor

113 Reconstruction up to a Scale Factor
Let It can be proved that

114 Reconstruction up to a Scale Factor
We have two choices of t, (t+ and t-) because of sign ambiguity and two choices of E, (E+ and E-). This gives us four pairs of translation vectors and rotation matrices.

115 Reconstruction up to a Scale Factor
Given and Construct the vectors w, and compute R Reconstruct the Z and Z’ for each point If the signs of Z and Z’ of the reconstructed points are both negative for some point, change the sign of and go to step 2. different for some point, change the sign of each entry of and go to step 1. both positive for all points, exit.

116

117