--------光流法 (Optical Flow) 第八章 基于运动视觉的稠密估计 --------光流法 (Optical Flow)

Presentation on theme: "--------光流法 (Optical Flow) 第八章 基于运动视觉的稠密估计 --------光流法 (Optical Flow)"— Presentation transcript:

1 --------光流法 (Optical Flow)
第八章 基于运动视觉的稠密估计 光流法 (Optical Flow)

2 稠密运动估计 2018/12/1 CV:Motion

3 光流法 运动场（motion field） 像素运动矢量 光流（optical flow） 图像亮度模式的表观运动 2018/12/1
CV:Motion

4 2018/12/1 CV:Motion

5 8.1 二维光流运动估计 根据图像亮度模式的变化估计物体的运动 1. 基于梯度的方法 (Gradient-based)
Differential Methods 2. 块匹配方法 (Block Matching) Region Matching 2018/12/1 CV:Motion

6 二维光流运动估计 关键性假设 假设各幅图像的亮度具有一致性 运动较小 Time = t Time = t+dt

7 8.1.1基于梯度的方法 基础性假设 点的亮度变化仅由运动引起 Taylor展开 光流约束方程
物理意义:如果一个固定的观察者观察一幅活动的场景，那么所得图象上某点灰度的(一阶)时间变化率是场景亮度变化率与该点运动速度的乘积。 2018/12/1 CV:Motion

8 Tracking in the 1D case: ?

9 Tracking in the 1D case: Temporal derivative Spatial derivative
Assumptions: Brightness constancy Small motion

10 Tracking in the 1D case: Iterating helps refining the velocity vector
Temporal derivative at 2nd iteration Can keep the same estimate for spatial derivative Converges in about 5 iterations

11 孔径问题(Aperture problem)

12 孔径问题(Aperture problem)
Motion along just an edge is ambiguous

13 孔径问题(Aperture problem)

14 孔径问题(Aperture problem)
需要足够的图像梯度信息，才能进行准确的估计 对于无纹理区域或者沿着边缘方向无法使用 只能在角点和纹理区域使用 图像中的每一点上有两个未知数u 和v ，但只有一个方程，因此，只使用一个点上的信息是不能确定光流的． 最优化问题,迭代求解 增加约束，使问题可解 2018/12/1 CV:Motion

15 基础假设： * Slide from Michael Black, CS

16 基础假设： * Slide from Michael Black, CS

17 From 1D to 2D tracking The Math is very similar: Aperture problem
Window size here ~ 11x11

18 求解孔径问题 How to get more equations for a pixel?
Basic idea: impose additional constraints most common is to assume that the flow field is smooth locally one method: pretend the pixel’s neighbors have the same (u,v) If we use a 5x5 window, that gives us 25 equations per pixel! * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

19 RGB version How to get more equations for a pixel?
Basic idea: impose additional constraints most common is to assume that the flow field is smooth locally one method: pretend the pixel’s neighbors have the same (u,v) If we use a 5x5 window, that gives us 25*3 equations per pixel! * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

20 Lukas-Kanade flow Prob: we have more equations than unknowns
Solution: solve least squares problem minimum least squares solution given by solution (in d) of: The summations are over all pixels in the K x K window This technique was first proposed by Lukas & Kanade (1981) described in Trucco & Verri reading * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

21 Conditions for solvability
Optimal (u, v) satisfies Lucas-Kanade equation When is This Solvable? ATA should be invertible ATA should not be too small due to noise eigenvalues l1 and l2 of ATA should not be too small ATA should be well-conditioned l1/ l2 should not be too large (l1 = larger eigenvalue) * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

22 Eigenvectors of ATA Suppose (x,y) is on an edge. What is ATA?
gradients along edge all point the same direction gradients away from edge have small magnitude is an eigenvector with eigenvalue What’s the other eigenvector of ATA? let N be perpendicular to N is the second eigenvector with eigenvalue 0 The eigenvectors of ATA relate to edge direction and magnitude Suppose M = vv^T . What are the eigenvectors and eigenvalues? Start by considering Mv * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

23 Edge large gradients, all the same large l1, small l2
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

24 Low texture region gradients have small magnitude small l1, small l2
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

25 High textured region gradients are different, large magnitudes
large l1, large l2 * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

26 Observation This is a two image problem BUT
Can measure sensitivity by just looking at one of the images! This tells us which pixels are easy to track, which are hard very useful later on when we do feature tracking... * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

27 误差分析 When our assumptions are violated
What are the potential causes of errors in this procedure? Suppose ATA is easily invertible Suppose there is not much noise in the image When our assumptions are violated Brightness constancy is not satisfied The motion is not small A point does not move like its neighbors window size is too large what is the ideal window size? * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

28 提高精度 Recall our small motion assumption It-1(x,y) It-1(x,y)
This is not exact To do better, we need to add higher order terms back in: It-1(x,y) This is a polynomial root finding problem Can solve using Newton’s method Also known as Newton-Raphson method Lukas-Kanade method does one iteration of Newton’s method Better results are obtained via more iterations * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

29 迭代更新 Iterative Lukas-Kanade Algorithm
Estimate velocity at each pixel by solving Lucas-Kanade equations Warp I(t-1) towards I(t) using the estimated flow field - use image warping techniques Repeat until convergence * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

30 降低分辨率 * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

31 Coarse-to-fine optical flow estimation
Gaussian pyramid of image It-1 Gaussian pyramid of image I image I image It-1 u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels image It-1 image I

32 Coarse-to-fine optical flow estimation
Gaussian pyramid of image It-1 Gaussian pyramid of image I image I image It-1 run iterative L-K warp & upsample run iterative L-K . image J image I

33 Multi-resolution Lucas Kanade Algorithm

34 (2) Horn-Schunck方法 增加全局平滑性约束（smoothness） 光流约束 全局平滑性约束 2018/12/1
CV:Motion

35 Horn-Schunck方法 用Gauss-Seidel方法迭代求解 优化目标 2018/12/1 CV:Motion

36 Horn-Schunck 方法 应用变分法求解全局能量优化问题

37 Horn-Schunck 方法

38 Horn-Schunck 方法 迭代方法 可以产生更高密度的运动场 可以填补无纹理区域 更容易受到噪声影响,需要做预处理

39 (3) Nagel方法 基于二阶导数的方法 面向平滑的约束，处理遮挡 Gauss-Seidel 迭代求解 2018/12/1
CV:Motion

40 2018/12/1 CV:Motion

41 Generalization * From Marc Pollefeys COMP

42 Generalization

43 * From Marc Pollefeys COMP 256 2003

44 * From Marc Pollefeys COMP 256 2003

45 Affine Flow * Slide from Michael Black, CS