Image Processing Models Inspired by Two Kinds of Double-Opponent Neurons in the Primary Visual Cortex Yong-Jie Li / 李永杰 School of Life Science and Technology Univ of Electronic Science & Tech of China liyj@uestc.edu.cn 电子科技大学生命科学与技术学院 / 神经信息教育部重点实验室

Outline Introduction Computational color constancy by double-opponent V1 cells with concentric center-surround receptive fields Contour detection by double-opponent V1 cells with oriented receptive fields Summary & Discussion

Outline Introduction Computational color constancy by double-opponent V1 cells with concentric center-surround receptive fields Contour detection by double-opponent V1 cells with oriented receptive fields Summary & Discussion

研究视觉的重要性 视觉系统是大脑最为重要的感知系统： 有70～80%的外界信息是由视觉系统接受和处理； 人脑皮层约50%以上的区域参与视觉信息的处理。 Where What V1

猴视觉皮层既平行又分级的30多个视觉皮层区域间连接的示意图 30多个视皮层区 10多个等级 300多条视觉联系

我们的目标 … 生物视觉系统是一种最高效、最合理的图像/视频信息处理系统。所以我们的目标—— 通过视觉神经机制的建模：阐明大脑视觉工作机理及发展类脑（Brain-like）技术。 阐明 大脑机理 发展 类脑技术 视觉系统 建模

视觉信息加工的基本单元 在视觉系统中，任何一级神经元在视网膜上都有一个对应的区域，该区域内的光学变化可以调制该神经元的反应，称为该神经元的感受野，或经典感受野（Classical RF, CRF）。

视觉信息加工的基本单元 + + 神经节、LGN、初级视皮层神经元的传统感受野（CRF）内的反应可被感受野外周区域中的刺激所调节 _ 外周

视觉信息加工的基本单元 神经元的(经典)感受野 + 感受野外周 经典感受野（classical receptive field, CRF） 非经典感受野（non-CRF, nCRF） （感受野外周、整合野） nCRF CRF + nCRF 组成了视觉信息加工的基本单元，为在更大范围内整合、加工视觉信息提供了可能。 CRF

Color is an important visual feature …

背侧流起始于V1，通过V2，进入背内侧区和中颞区（MT，亦称V5），然后抵达顶下小叶。背侧流常被称为“空间通路”（Where pathway），参与处理物体的空间位置信息以及相关的运动控制，例如眼跳（saccade）和伸取（Reaching）。 腹侧流起始于V1，依次通过V2，V4，进入下颞叶（Inferior temporal lobe）。该通路常被称为“内容通路”（What pathway），参与物体识别，例如面孔识别。该通路也于长期记忆有关。 --------------- 颞下回后区（PIT） （Conway, 2009）

Retina

Ganglion cells M cell P cell K cell

M cell P cell K cell

Color-opponent cells L+ M-

Single- & double-opponent cells Single-opponency double-opponency RG, LGN, V1 V1

Single- & double-opponent Single-opponent Single- & double-opponent

Single-opponency double-opponency

Luminance cells (60%) Color-luminance cells (29%) Color cells (11%)

color-luminance cells Color cells Proportion 60% (100/167) 29% (48/167) 11% (19/167) Opponent Non-opponent Double-opponent Single-opponent & Spatial tuning Band-pass Low-pass Response Achromatic boundaries Achromatic & chromatic boundaries Chromatic regions Orientation tuning Orientation-selectivity Non-orientation-selectivity Location 4B,4Cα,5,6 2/3 2/3,5,6 Johnson (2001) Nature Neuroscience; Solomon (2007) Nature Neuroscience

Luminance cells (60%) color-luminance cells(29%) Color cells(11%) Non-opponent Double-opponent Single-opponent Simple cell Complex cell few

We are concerned with the functional roles of the two types of double-opponent V1 cells

Outline Introduction Computational color constancy by double-opponent V1 cells with concentric center-surround receptive fields Contour detection by double-opponent V1 cells with oriented receptive fields Summary & Discussion

Color constancy

Color constancy The effect that the perceived color of a surface remains constant despite changes in the intensity and spectral composition of the illumination. Color constancy has had a long history of analysis (Monge, 1789; Young, 1807; von Helmholtz, 1867; Hering, 1920; …)

V4 ?

Single-opponent Input from LGN Double-opponent V1

(ICCV, 2013(oral); IEEE Trans PAMI, 2015)

(Reinhard et al. 2001)

(Ebner, 2007)

思路：高级视皮层神经元利用初级视皮层神经元的输出估计场景光源的颜色，然后从色偏图像中消除此光源的影响，恢复出场景物体的真实颜色。

Results on Gehler-Shi dataset (Lower angular error is better).

Median angular errors of various models on the Gehler-Shi dataset with 568 high dynamic range linear images, including a variety of indoor and outdoor scenes

Median angular errors of different algorithms on SFU lab dataset, which contains 321 available images of 31 different objects captured with calibrated camera under 11 different lights in laboratory.

Why the model works? 综合利用了边缘信息和部分区域信息

小 结 生理研究上，提出了与之前完全不同的推论—V1区的特定类型的双拮伉神经元输出编码了场景光源；此光源进一步地被更高级的视皮层用来矫正场景颜色，得到物体的真实物理颜色。 工程应用上，得到了与目前最好方法效果相当的结果，但方法简单，计算效率高。 下一步：多光源。

Outline Introduction Computational color constancy by double-opponent V1 cells with concentric center-surround receptive fields Contour detection by double-opponent V1 cells with oriented receptive fields Summary & Discussion

Oriented double-opponent cells in V1 (equal cone inputs) (unequal cone inputs)

V1 Double-Opponent (Output Layer) LGN Single-Opponent (Middle Layer) Retina Cone (Input Layer) CVPR, 2013; IEEE Trans IP, 2015

Input image R G B (Berkley Dataset)

R-G Channel B-Y Channel Luminance edge

Original image Ground-truth

Pb [3] Our model [3] Martin, et al., IEEE Trans. on PAMI, vol. 26, pp. 530-549, 2004.

(Computation time (s) per image averaged over 100 images)

Robert Shapley, Michael J. Hawken, Vision Research. 2011.  Image Fig. 5. The average spatial frequency tuning for three populations of V1 neurons. The tuning functions were estimated by Schluppeck and Engel (2002) from 230 neurons recorded by Johnson, Hawken, and Shapley. The dotted line represents the responses of the color-preferring neurons. It shows the characteristic low-pass spatial frequency tuning reported in most studies (Johnson et al., 2001, 2004, 2008; Lennie et al., 1990; Solomon et al., 2004; Thorell et al., 1984). The dashed line shows the responses of color-luminance neurons: cells classified as having robust responses to equiluminant color and to black/white luminance when the stimuli are matched for cone contrast. Most of the chromatically opponent color-luminance simple cells are double-opponent in that they have spatially separated chromatically opponent responses to L- and M-cones (Johnson et al., 2008; Fig. 6). The solid line is the average spatial frequency tuning of the luminance-preferring neurons. The maximum responses of luminance cells to luminance patterns are more than twice the amplitude of the best response to equiluminance. The tuning of the colorluminance and luminance cells are bandpass and similar in both preferred spatial frequency (2.56 ?1.26 cy/deg and 2.09 ?1.00 cy/deg respectively) and in bandwidth (2.05 ?0.70 octaves (full width, half height) and 2.96 ?0.69 octaves respectively). Robert Shapley, Michael J. Hawken, Vision Research. 2011. 

Outline Introduction Computational color constancy by double-opponent V1 cells with concentric center-surround receptive fields Contour detection by double-opponent V1 cells with oriented receptive fields Summary & Discussion

V1 cells with different receptive field (RF) properties have different functional roles: DO V1 cells with concentric center-surround RFs can code the illuminant color, which is used by higher area (e.g., V4) to realize color constancy. DO V1 cells with oriented RFs and unbalanced cone inputs can response to (detect) both the color- and luminance-defined contours.

Related papers [1] Kaifu Yang, Shaobing Gao, Chaoyi Li, Yongjie Li*, Efficient Color Boundary Detection with Color-opponent Mechanisms, CVPR, 2013, pp2810-2817. [2] Shaobing Gao, Kaifu Yang, Chaoyi Li, Yongjie Li*, A Color Constancy Model with Double-Opponency Mechanisms, ICCV, 2013, pp929-936 (oral paper). [3] Shaobing Gao, Kaifu Yang, Chaoyi Li, Yongjie Li*, Color Constancy Using Double-Opponency, IEEE Transactions on PAMI, 2015 (DOI: 10.1109/TPAMI.2015.2396053) [4] Kai-Fu Yang, Shao-Bing Gao, Ce-Feng Guo, Chao-Yi Li, and Yong-Jie Li*, Boundary Detection Using Double-Opponency and Spatial Sparseness Constraint, IEEE Transactions on Image Processing, 2015, 24(8): 2565-2578.

Thanks to … 李朝义院士、颜红梅教授、王玲副教授 … 博士生：杨开富、高绍兵、张显石 … 课题组网站：www.neuro.uestc.edu.cn/vccl

谢 谢 ! 敬请批评指正 ！ Code available at: http://www.neuro.uestc.edu.cn/vccl/ liyj@uestc.edu.cn