（A Survey on Image-based Rendering Techniques）

（A Survey on Image-based Rendering Techniques）
计算机图形学学科研讨会基于图像绘制（IBR）技术综述（A Survey on Image-based Rendering Techniques）石教英浙江大学计算机学院计算机辅助设计与图形学国家重点实验室 2005年5月

目录一、基于图像绘制（IBR）技术定义二、典型IBR技术演示三、IBR技术理论基础-Plenoptic Funciton
目录一、基于图像绘制（IBR）技术定义二、典型IBR技术演示 1、Panorama Mosaics 2、Tour-Into-Picture 3、Light Field 4、Feature-based Morphing 三、IBR技术理论基础-Plenoptic Funciton 1、An introduction 2、how to handle with plenoptic fuction 四、IBR技术发展 1、Image Matting 2、Digital PhotoMontage 3、High-Dynamic-Range Image Display 4、Plenoptic photography 五、基于图像绘制技术定义的扩展

一、基于图像绘制（IBR）技术定义 Traditional Computer Graphics
Input: Geometry, Material Properties (Color, Reflectance,…etc.), Lighting. Transformation and Rasterization. Computer Vision methods to recover models. Transform (& Lighting) Rasterization

一、基于图像绘制（IBR）技术定义 Top Level Survey 3D Graphics Image-Based Rendering
Geometry or Surface Based Rendering & Modeling Sample-Based Graphics Image-Based Rendering & Modeling Volume Rendering

一、基于图像绘制（IBR）技术定义

一、基于图像绘制（IBR）技术定义 Why IBR? Problems of triangle-based graphics:
Always starts from scratch. Millions of sub-pixel triangles. Geometry IBR Modeling Difficult Easy Complexity #triangles #pixels Fidelity Synthetic Acquired

一、基于图像绘制（IBR）技术定义 Definition of IBR (by Sing Bing Kang) image-based rendering techniques rely on interpolation using the original set of input images or pixel reprojection from source images onto the target image in order to produce a novel virtual view

一、基于图像绘制（IBR）技术定义 Definition of IBR (by Cha Zhang & Tsuhan Chen of CMU) Given a continuous plenoptic function that describes a scene, image-based rendering is a process of two stages: – sampling and rendering. In the sampling stage, samples are taken from the plenoptic function for representation and storage. In the rendering stage, the continuous plenoptic function is reconstructed with the captured samples.

二、典型IBR技术演示 Paronoma Mosaics Demo

二、典型IBR技术演示 Tour-Into-Picture

二、典型IBR技术演示 Lightfield Video

二、典型IBR技术演示 Feature-based Morphing

三、IBR技术理论基础-Plenoptic Funciton
1. An Introduction Two ways of describing the world: A source description: ——The world can be described by geometric models, texture maps, refection models, lighting and shading models. An appearance description: ——The world can be described by the dense array of light rays filling the space, which can be observed by posing eyes or cameras in the space. The light rays can be represented through the Plenoptic function. The traditional model-based rendering approach adopts the source description method. The image-based rendering approach adopots the plenoptic function to describe the world.

IBR: An Old Story Plenoptic function As pointed out by Adelson and Bergen (1991): The world is made of three-dimensional objects, but these objects do not communicate their properties directly to an observer. … The plenoptic function serves as the sole communication link between the physical objects and their corresponding retinal images. It is the intermediary between the world and the eye.

7D Plenoptic Function

2. How to Handle Plenoptic Function? Two Stages sampling and reconstruction of sampled signals Two directions for simplication Restrain the viewing space of the viewers View point Perceptual Introduce some source descriptions into IBR Geometry Depth

Restraining in View Space Assumption 1: Wavelength Constant wavelength RGB Almost all the practical representations of IBR make this assumption.

Restraining in View Space Assumption 2: Air Air is translucent Radiances along a light ray through empty space remain constant.

Restraining in View Space Assumption 3: Time Static scene Images captured at different time and positions can be used together to render novel views. Too large size for dynamic scene

Restraining in View Space Assumption 4: Viewpoint The viewer is constrained to be on a surface Acceptable: Human eyes are usually at a certain height-level Human eyes are less sensitive to vertical parallax and lighting changes

Restraining in View Space Assumption 5: Viewpoint The viewer moves along a certain path. Reduces 2 dimensions from the full plenoptic function. Too large size for dynamic scene

Restraining in View Space Assumption 6: Viewpoint The viewer has a fixed position. Reduces the dimension of the plenoptic function by 3. No 3D effects can possibly be perceived. Similarity to regular images and videos.

Various Representations

6D – The Surface Plenoptic Function Assumption 2. As we discussed, when radiance along a light ray through empty space remains constant. 6D Position on the surface (2D) Light ray direction(2D) Time (1D) and wavelength (1D).

Example The surface light field could be considered as dimension-reduced version of SPF. D. N. Wood, D. I. Azuma, K. Aldinger, B. Curless, T. Duchamp, D. H. Salesin and W. Stuetzle, “Surface light fields for 3D photography”, Computer Graphics (SIGGRAPH’00), July 2000.

5D – Plenoptic Modeling and Light field Video Plenoptic modeling, which is a 5D function 3D for the camera position, 2D for the cylindrical image. L. McMillan and G. Bishop, “Plenoptic modeling: an image-based rendering system”, Computer Graphics (SIGGRAPH’95), August 1995,

Plenoptic Modeling To render a novel view from the 5D representation, the close-by cylindrical projected images are warped to the viewing position based on their epipolar relationship and some visibility tests.

4D – Light field / Lumigraph Assumption 1, 2 and 3 Trickily parameterize ray space to 2-plane representation.

3D – Concentric Mosaics Assumption 1, 2, 3 and 4

Camera Moving on a Plane Center camera => panorama Off-centered cameras => motion parallax Camera Beam Tripod

Multi-perspective Image Top view Manifold mosaic

Panorama from Slit Images Single perspective point …...

Concentric Mosaics Mosaics at concentric circles

Concentric Mosaics Render a novel image Top view

2D – Image Mosaicing Composes one single mosaic with multiple input images In most cases, the light rays recorded in the mosaic share the same center-of-projection (COP) -Panoramic mosaic or panorama

Examples

2D – Image Mosaicing In the more general scenario, the cameras of the input images can move in free form and the resultant mosaic has MCOPs Manifold Mosaic.

Examples

Panoramic Cameras Rotating Cameras Kodak Cirkut Globuscope Stationary Cameras Be Here

Quicktime VR Using environmental maps Cylindrical Cubic spherical At a fixed point, sample all the ray directions. Users can look in both horizontal and vertical directions

Mars Pathfinder Panorama

IBR with Various Source Descriptions

Correspondence between Images

Example 1: View Morphing (SIGGRAPH 96) Steve Seitz etc.

Dense Depth Map

Various Rendering Techniques Polygon Camera Planar Sprites LDI Sprite with Depth Viewing Region Environment Map

Layered Depth Images Idea: Handle disocclusion Store invisible geometry in depth images

Texture Map (+ Scene Geometry) Image based modeling View dependent texture mapping

Reflection Models (+ Scene Geometry) Image based relighting

Example: Image Based Relighting Siggraph Relighting.mov

Turn to another question… What is the relationship between geometry and image based rendering? J. -X. Chai, X. Tong, S. –C. Chan, and H. –Y. Shum. Plenopic sampling. In Proc. SIGGRAPH, 2000

四、IBR技术发展 Digital image matting Interactive digital photomontage High-dynamic range image processing Light field photograph

四、IBR技术发展——Image matting
什么是抠图？图像抽取的简称，是指把任意形状的前景物体从图像中抽取出来的一种技术。它与图像合成是一对互逆的操作。 [Porter84]

一般来说，自然图像抠图可以分为区域分割、颜色估计与alpha估计三个步骤。主要存在两个问题：计算前景与背景颜色分量的过程复杂，导致速度变慢 alpha估计方法不够鲁棒，以至抠图效果欠佳

区域划分抠图是个不可精确求解问题，对图像施加约束或提供人工交互是必然的通过画前景和背景两条轮廓线把图像划分为前景、背景和未知三个区域

颜色估计

alpha估计(1) RGB颜色空间中的alpha估计三通道加权法(Knockout) 投影法(Ruzon00,Hillman01,Chuang01)

alpha估计(2) RGB颜色空间中的alpha估计

alpha估计(3) 感知颜色空间中的alpha估计

结果与比较(1) Bayesian Global Poisson 我们的方法

结果与比较(2) Bayesian Global Poisson 我们的方法

结果与比较(3)

“GrabCut” —— Interactive Foreground Extraction using Iterated Graph Cuts Carsten rother, Vladimir Kolmogorov, Andrew Blake

Lazy Snapping Yin Li, Jian Sun, Chi-Keung Tang, Heung-Yeung Shum

四、IBR技术发展——Interactive Digital Photomontage
An interactive computer-assisted framework for combining parts of a set of photographs into a single composite picture Two primary techniques Graph-cut optimization Gradient-domain fusion

四、IBR技术发展——Interactive Digital Photomontage

四、IBR技术发展——High-Dynamic Range Image Processing
如何在低动态范围的显示设备，如100：1的CRT上显示高动态范围（如25000：1）的图像构造HDR Radiance Map，压缩之，显示在普通显示设备上获得HDR Radiance Map的方法拍几张曝光度不同的照片拍全景视频

多次曝光法生成HDR radiance map

用全景视频生成HDR radiance map

LDR图像增强

四、IBR技术发展——Light Field Photography
Plenoptic camera

292*292微镜图像构成完整的Light field图像。这些微镜图像反应出真实世界中光线的结构，从而包含了对象的深度信息。
左一，是焦平面前左二，是焦平面后左三，是焦平面上

四、IBR技术发展——Light Field Photography

五、基于图像绘制技术定义的扩展基于图像的绘制（IBR）是一类产生新的视觉效果的图像采样和绘制技术。新的视觉效果包括：合成新视点和视角的视图（Paranoma mosaic, CM, Light field, View morphing, TIP），新的光照效果（IB relighting），合成新的图像（Image matting），交互式数字蒙太奇效果（Interactive digital photomontage ），扩大图像的动态深度（HDR），全光照相术（Light field photography），综合孔径共焦成像技术（Synthetic aperture confocal imaging）等效果。采样图像可以是真实世界的照相，也可以是计算机综合的图形。处理时可以完全不需要采样图像所含的辅助信息，也可以辅以一定的采样图像的几何、光照和纹理等信息，构成不同类型的IBR技术。

谢谢

（A Survey on Image-based Rendering Techniques）

Similar presentations

Presentation on theme: "（A Survey on Image-based Rendering Techniques）"— Presentation transcript:

Similar presentations

About project

反馈

请登录

Auth with social network:

（A Survey on Image-based Rendering Techniques）

Similar presentations

Presentation on theme: "（A Survey on Image-based Rendering Techniques）"— Presentation transcript:

Similar presentations

About project

反馈