（A Survey on Image-based Rendering Techniques） 计算机图形学学科研讨会 基于图像绘制（IBR）技术综述 （A Survey on Image-based Rendering Techniques） 石教英 浙江大学计算机学院 计算机辅助设计与图形学国家重点实验室 2005年5月

目 录 一、基于图像绘制（IBR）技术定义 二、典型IBR技术演示 三、IBR技术理论基础-Plenoptic Funciton 目 录 一、基于图像绘制（IBR）技术定义 二、典型IBR技术演示 1、Panorama Mosaics 2、Tour-Into-Picture 3、Light Field 4、Feature-based Morphing 三、IBR技术理论基础-Plenoptic Funciton 1、An introduction 2、how to handle with plenoptic fuction 四、IBR技术发展 1、Image Matting 2、Digital PhotoMontage 3、High-Dynamic-Range Image Display 4、Plenoptic photography 五、基于图像绘制技术定义的扩展

一、基于图像绘制（IBR）技术定义 Traditional Computer Graphics Input: Geometry, Material Properties (Color, Reflectance,…etc.), Lighting. Transformation and Rasterization. Computer Vision methods to recover models. Transform (& Lighting) Rasterization

一、基于图像绘制（IBR）技术定义 Top Level Survey 3D Graphics Image-Based Rendering Geometry or Surface Based Rendering & Modeling Sample-Based Graphics Image-Based Rendering & Modeling Volume Rendering

一、基于图像绘制（IBR）技术定义

一、基于图像绘制（IBR）技术定义 Why IBR? Problems of triangle-based graphics: Always starts from scratch. Millions of sub-pixel triangles. Geometry IBR Modeling Difficult Easy Complexity #triangles #pixels Fidelity Synthetic Acquired

一、基于图像绘制（IBR）技术定义 Definition of IBR (by Sing Bing Kang) image-based rendering techniques rely on interpolation using the original set of input images or pixel reprojection from source images onto the target image in order to produce a novel virtual view

一、基于图像绘制（IBR）技术定义 Definition of IBR (by Cha Zhang & Tsuhan Chen of CMU) Given a continuous plenoptic function that describes a scene, image-based rendering is a process of two stages: – sampling and rendering. In the sampling stage, samples are taken from the plenoptic function for representation and storage. In the rendering stage, the continuous plenoptic function is reconstructed with the captured samples.

二、典型IBR技术演示 Paronoma Mosaics Demo

二、典型IBR技术演示 Tour-Into-Picture

二、典型IBR技术演示 Lightfield Video

二、典型IBR技术演示 Feature-based Morphing

三、IBR技术理论基础-Plenoptic Funciton 1. An Introduction Two ways of describing the world: A source description: ——The world can be described by geometric models, texture maps, refection models, lighting and shading models. An appearance description: ——The world can be described by the dense array of light rays filling the space, which can be observed by posing eyes or cameras in the space. The light rays can be represented through the Plenoptic function. The traditional model-based rendering approach adopts the source description method. The image-based rendering approach adopots the plenoptic function to describe the world.

三、IBR技术理论基础-Plenoptic Funciton IBR: An Old Story Plenoptic function As pointed out by Adelson and Bergen (1991): The world is made of three-dimensional objects, but these objects do not communicate their properties directly to an observer. … The plenoptic function serves as the sole communication link between the physical objects and their corresponding retinal images. It is the intermediary between the world and the eye.

三、IBR技术理论基础-Plenoptic Funciton 7D Plenoptic Function

三、IBR技术理论基础-Plenoptic Funciton 2. How to Handle Plenoptic Function? Two Stages sampling and reconstruction of sampled signals Two directions for simplication Restrain the viewing space of the viewers View point Perceptual Introduce some source descriptions into IBR Geometry Depth

三、IBR技术理论基础-Plenoptic Funciton Restraining in View Space Assumption 1: Wavelength Constant wavelength RGB Almost all the practical representations of IBR make this assumption.

三、IBR技术理论基础-Plenoptic Funciton Restraining in View Space Assumption 2: Air Air is translucent Radiances along a light ray through empty space remain constant.

三、IBR技术理论基础-Plenoptic Funciton Restraining in View Space Assumption 3: Time Static scene Images captured at different time and positions can be used together to render novel views. Too large size for dynamic scene

三、IBR技术理论基础-Plenoptic Funciton Restraining in View Space Assumption 4: Viewpoint The viewer is constrained to be on a surface Acceptable: Human eyes are usually at a certain height-level Human eyes are less sensitive to vertical parallax and lighting changes

三、IBR技术理论基础-Plenoptic Funciton Restraining in View Space Assumption 5: Viewpoint The viewer moves along a certain path. Reduces 2 dimensions from the full plenoptic function. Too large size for dynamic scene

三、IBR技术理论基础-Plenoptic Funciton Restraining in View Space Assumption 6: Viewpoint The viewer has a fixed position. Reduces the dimension of the plenoptic function by 3. No 3D effects can possibly be perceived. Similarity to regular images and videos.

三、IBR技术理论基础-Plenoptic Funciton Various Representations

三、IBR技术理论基础-Plenoptic Funciton 6D – The Surface Plenoptic Function Assumption 2. As we discussed, when radiance along a light ray through empty space remains constant. 6D Position on the surface (2D) Light ray direction(2D) Time (1D) and wavelength (1D).

三、IBR技术理论基础-Plenoptic Funciton Example The surface light field could be considered as dimension-reduced version of SPF. D. N. Wood, D. I. Azuma, K. Aldinger, B. Curless, T. Duchamp, D. H. Salesin and W. Stuetzle, “Surface light fields for 3D photography”, Computer Graphics (SIGGRAPH’00), July 2000.

三、IBR技术理论基础-Plenoptic Funciton 5D – Plenoptic Modeling and Light field Video Plenoptic modeling, which is a 5D function 3D for the camera position, 2D for the cylindrical image. L. McMillan and G. Bishop, “Plenoptic modeling: an image-based rendering system”, Computer Graphics (SIGGRAPH’95), August 1995,

三、IBR技术理论基础-Plenoptic Funciton Plenoptic Modeling To render a novel view from the 5D representation, the close-by cylindrical projected images are warped to the viewing position based on their epipolar relationship and some visibility tests.

三、IBR技术理论基础-Plenoptic Funciton 4D – Light field / Lumigraph Assumption 1, 2 and 3 Trickily parameterize ray space to 2-plane representation.

三、IBR技术理论基础-Plenoptic Funciton

三、IBR技术理论基础-Plenoptic Funciton 3D – Concentric Mosaics Assumption 1, 2, 3 and 4

三、IBR技术理论基础-Plenoptic Funciton Camera Moving on a Plane Center camera => panorama Off-centered cameras => motion parallax Camera Beam Tripod

三、IBR技术理论基础-Plenoptic Funciton Multi-perspective Image Top view Manifold mosaic

三、IBR技术理论基础-Plenoptic Funciton Panorama from Slit Images Single perspective point …...

三、IBR技术理论基础-Plenoptic Funciton Concentric Mosaics Mosaics at concentric circles

三、IBR技术理论基础-Plenoptic Funciton Concentric Mosaics Render a novel image Top view

三、IBR技术理论基础-Plenoptic Funciton 2D – Image Mosaicing Composes one single mosaic with multiple input images In most cases, the light rays recorded in the mosaic share the same center-of-projection (COP) -Panoramic mosaic or panorama

三、IBR技术理论基础-Plenoptic Funciton Examples

三、IBR技术理论基础-Plenoptic Funciton 2D – Image Mosaicing In the more general scenario, the cameras of the input images can move in free form and the resultant mosaic has MCOPs Manifold Mosaic.

三、IBR技术理论基础-Plenoptic Funciton Examples

三、IBR技术理论基础-Plenoptic Funciton Panoramic Cameras Rotating Cameras Kodak Cirkut Globuscope Stationary Cameras Be Here

三、IBR技术理论基础-Plenoptic Funciton Quicktime VR Using environmental maps Cylindrical Cubic spherical At a fixed point, sample all the ray directions. Users can look in both horizontal and vertical directions

三、IBR技术理论基础-Plenoptic Funciton Mars Pathfinder Panorama

三、IBR技术理论基础-Plenoptic Funciton IBR with Various Source Descriptions

三、IBR技术理论基础-Plenoptic Funciton Correspondence between Images

三、IBR技术理论基础-Plenoptic Funciton Example 1: View Morphing (SIGGRAPH 96) Steve Seitz etc.

三、IBR技术理论基础-Plenoptic Funciton Dense Depth Map

三、IBR技术理论基础-Plenoptic Funciton Various Rendering Techniques Polygon Camera Planar Sprites LDI Sprite with Depth Viewing Region Environment Map

三、IBR技术理论基础-Plenoptic Funciton Layered Depth Images Idea: Handle disocclusion Store invisible geometry in depth images

三、IBR技术理论基础-Plenoptic Funciton Texture Map (+ Scene Geometry) Image based modeling View dependent texture mapping

三、IBR技术理论基础-Plenoptic Funciton Reflection Models (+ Scene Geometry) Image based relighting

三、IBR技术理论基础-Plenoptic Funciton Example: Image Based Relighting Siggraph Relighting.mov

三、IBR技术理论基础-Plenoptic Funciton Turn to another question… What is the relationship between geometry and image based rendering? J. -X. Chai, X. Tong, S. –C. Chan, and H. –Y. Shum. Plenopic sampling. In Proc. SIGGRAPH, 2000

四、IBR技术发展 Digital image matting Interactive digital photomontage High-dynamic range image processing Light field photograph

四、IBR技术发展——Image matting 什么是抠图？ 图像抽取的简称，是指把任意形状的前景物体从图像中抽取出来的一种技术。它与图像合成是一对互逆的操作。 [Porter84]

四、IBR技术发展——Image matting

四、IBR技术发展——Image matting

四、IBR技术发展——Image matting 一般来说，自然图像抠图可以分为区域分割、颜色估计与alpha估计三个步骤。主要存在两个问题： 计算前景与背景颜色分量的过程复杂，导致速度变慢 alpha估计方法不够鲁棒，以至抠图效果欠佳

四、IBR技术发展——Image matting 区域划分 抠图是个不可精确求解问题，对图像施加约束或提供人工交互是必然的 通过画前景和背景两条轮廓线把图像划分为前景、背景和未知三个区域

四、IBR技术发展——Image matting 颜色估计

四、IBR技术发展——Image matting alpha估计(1) RGB颜色空间中的alpha估计 三通道加权法(Knockout) 投影法(Ruzon00,Hillman01,Chuang01)

四、IBR技术发展——Image matting alpha估计(2) RGB颜色空间中的alpha估计

四、IBR技术发展——Image matting alpha估计(3) 感知颜色空间中的alpha估计

四、IBR技术发展——Image matting 结果与比较(1) Bayesian Global Poisson 我们的方法

四、IBR技术发展——Image matting 结果与比较(2) Bayesian Global Poisson 我们的方法

四、IBR技术发展——Image matting 结果与比较(3)

四、IBR技术发展——Image matting “GrabCut” —— Interactive Foreground Extraction using Iterated Graph Cuts Carsten rother, Vladimir Kolmogorov, Andrew Blake

四、IBR技术发展——Image matting

四、IBR技术发展——Image matting Lazy Snapping Yin Li, Jian Sun, Chi-Keung Tang, Heung-Yeung Shum

四、IBR技术发展——Image matting

四、IBR技术发展——Interactive Digital Photomontage An interactive computer-assisted framework for combining parts of a set of photographs into a single composite picture Two primary techniques Graph-cut optimization Gradient-domain fusion

四、IBR技术发展——Interactive Digital Photomontage

四、IBR技术发展——High-Dynamic Range Image Processing 如何在低动态范围的显示设备，如100：1的CRT上显示高动态范围（如25000：1）的图像 构造HDR Radiance Map，压缩之，显示在普通显示设备上 获得HDR Radiance Map的方法 拍几张曝光度不同的照片 拍全景视频

四、IBR技术发展——High-Dynamic Range Image Processing

四、IBR技术发展——High-Dynamic Range Image Processing 多次曝光法生成HDR radiance map

四、IBR技术发展——High-Dynamic Range Image Processing 用全景视频生成HDR radiance map

四、IBR技术发展——High-Dynamic Range Image Processing LDR图像增强

四、IBR技术发展——Light Field Photography Plenoptic camera

四、IBR技术发展——Light Field Photography Plenoptic camera

292*292微镜图像构成完整的Light field图像。这些微镜图像反应出真实世界中光线的结构，从而包含了对象的深度信息。 左一，是焦平面前 左二，是焦平面后 左三，是焦平面上

四、IBR技术发展——Light Field Photography

四、IBR技术发展——Light Field Photography

四、IBR技术发展——Light Field Photography

五、基于图像绘制技术定义的扩展 基于图像的绘制（IBR）是一类产生新的视觉效果的图像采样和绘制技术。新的视觉效果包括：合成新视点和视角的视图（Paranoma mosaic, CM, Light field, View morphing, TIP），新的光照效果（IB relighting），合成新的图像（Image matting），交互式数字蒙太奇效果（Interactive digital photomontage ），扩大图像的动态深度（HDR），全光照相术（Light field photography），综合孔径共焦成像技术（Synthetic aperture confocal imaging）等效果。采样图像可以是真实世界的照相，也可以是计算机综合的图形。处理时可以完全不需要采样图像所含的辅助信息，也可以辅以一定的采样图像的几何、光照和纹理等信息，构成不同类型的IBR技术。

谢 谢