Enhancement Algorithms in Digital Camera 勤益科技大學資訊工程系 智慧生活實驗室 林灶生

Outline 1 Introduction 2 Color Balance 3 Luminance & Contrast Compensation Conclusion 4

Introduction Luminance Chroma Key tone Color temperature Contrast Color cast Luminance Key tone Contrast 基本上, 光是具有不同波長的電磁波,而波長在700-400奈米之間的光，恰好能為人眼的視神經所感知，故稱為可見光，亦即一般人所稱的光，人眼及腦部除了會對光的強弱產生明暗的感覺，亦會對不同波長的光產生不同的色彩的感覺。 而這些光的特質被照相機忠實的記錄下來，故數位影像會產生色偏及色調對比的問題。 日常所見的光多半是由多種波長的光混合而成，不同的混合比率會呈現出不同的色彩。科學上稱一個光源所含的各種波長的強度分佈(即混合比率)為該光源的「光譜(Spectrum)」，在儀器所測到的叫「光譜」，由眼睛看到就是「色彩」。

Color Balance

Color Balance in Camera 單眼像機中的白平衡設定

Color Balance Setup 在陰天時於室外拍攝 由於是陰天，所以在陰天模式下所拍攝的影像色感會是最正常的。反之，色溫設定值比陰天模式6000K更高的陰影模式就會呈現紅色，色溫設定值較低的鎢絲燈模式等狀態下則會呈現藍色。 色偏 相機會忠實地反應不同色溫所造成的「染色」結果，使得整張照片顏色很一致地傾向某種顏色。色偏的原因是相機的白平衡自動偵測失誤或設定模式不對所導致。 色彩平衡(白平衡) 白平衡是修正色偏機制 目的：因為人的眼睛和大腦會自動校正環境光源的差異， 而相機無法做到。故為了要讓相片的顏色與人眼所認知的一致，就必須在攝影過程中，修正環境光線中所造成的細微影響。

Color Temperature black Body radiator Hue of light Kelvin (K) temperature Cold Tone Warm Tone 「色溫」是科學上一種量測溫度的方法。在物理上，物體受熱就會開始發光，在絕對溫度0(0oK)度時任何物體都是純黑的，隨著溫度的上升物體會散發出不同顏色的光，一般是先紅再逐漸變藍，也就是說隨著溫度的變化，物體發光的顏色(即光譜)也會變化。

Color Cast Daylight ≒ 5500K R ≒ G ≒ B Cold Tone > 5500K  R < B 正常的色溫就是人類眼睛最熟悉的太陽光，此時色溫就是5500K。把5500K的日光經過三稜鏡分光，會產生紅橙黃綠藍…等色彩，其中的主色是RGB為三原色 (1) 白光的色溫約為5500K，此時RGB的佔比相近，所以被攝體的顏色會被忠實的反射 (2) 若某光源的色溫較高(大於5500K)，例如B的比例較R高，所以紅色物體會變暗，白色物體因為反射藍光較多而呈偏藍色 (3) 若某光源的色溫較低(小於5500K)，例如R的比例較B高，此時紅色物體較藍色物體反射更多光線，而原本白色物體也會因為反射紅光較多的緣故而呈偏紅或偏黃。 Daylight ≒ 5500K R ≒ G ≒ B Cold Tone > 5500K  R < B Warm Tone < 5500K  R > B

Color Balance Algorithms

Gray World Assumption The average value of the RED, GREEN, and BLUE components of the image with sufficient amount of color variations should average out to a common gray value. Gray World Assumption 當一張色彩變化豐富的影像被呈現時，GWA認為這影像所組成的RGB平均值，應該綜合平均出一個理想的灰色值（Chikane & Fun, 2006）。於是，GWA根據此一理想灰色值，分別動態調整RGB的灰階分布圖之後，再還原重建該影像的應有色彩。但是，GWA有一個很大的缺點，當場景中有著顯著的主調色彩（例如：紅色調）時，那麼調整之後的影像則有嚴重偏色的情形。

Retinex Edwin H. Land et al in 1971 Retinex=Retina+ Cortex I(x,y)=S(x,y)* R(x,y)  R’(x,y) = log(I(x,y) – log(S(x,y)) Sharpening、 Color constancy 、dynamic range Scene Reflectance (R) Intensity(I) Source Illumination(S) I(x ,y)=S(x, y)*R(x ,y) Retinex Retinex=Retina(視網膜) + Cortex(大腦皮層) 理論主要包含兩部分 物體的顏色是由物體對長波(R)、中波(G)和短波(B)光線的反射能力決定，而不是由反射光強度的絕對值決定 物體的色彩不受光照非均性的影響，具有一致性色彩一致性(顏色恒常性) 理論的基本思想：就是將原始圖像看成是由「光源照入所能看見景物的量(Source illumination)」和「場景中物體反射光的量(Object reflectance)」組成，Source illumination直接決定一幅圖像中像素能夠達到的動態範圍，Object reflectance決定了圖像的內在性質，因此，在原始圖像中去除或降低Source illumination的影響從而保留Object reflectance是Retinex理論的基本思想。 **首先將待增強的圖像看成是由反射光分量和入射光分量兩部分的乘積，其中反射分量對應於圖像的本來面貌，入射分量對應於圖像中的干擾部分．然後採用某種途徑估算出入射分量，並從待增強圖像中除去入射分量，得到反射分量部分，還原圖像的本來面貌，實現對圖像的增強。 **用對數計算，主要原因一方面可以簡化計算，將乘法運算轉變為加法運算，另一方面也符合人眼的非線性特性。 特點：Retinex可在動態範圍壓縮、邊緣增強和顏色恒常三方面達到平衡

Retinex Jobson et. al. Single Scale Retinex (SSR) Multi-Scale Retinex (MSR) Multi-Scale Retinex with Color Restoration (MSRCR) Gaussian function 在Retinex算 法的發展史中，曾經出現過平方反比的環繞形式、指數形式以及高斯指數形式。 SSR Jobson論證了Gaussian Function可以對原圖像提供更局部的準確處理，因而可以更好地增強圖像。 高斯函數的標準差σ 稱之Retinex的尺度參數。 (1) 當σ值較大時，所估計的入射分量較平滑，得到的增強圖像色彩保真能力好，但是細節增強效果較差。 (2) 當σ值較小時，所估計的入射分量比較崎嶇，實現圖像的動態範圍壓縮，得到的增強圖像細節突出，但是色彩保真效果較差。 故SSR無法同時達到保證細節增強與顏色保真的效果。 MSR MSR是將不同尺度σ 增強過的圖像利用線性加權方法組合起來，實現色彩保真和細節增強之間的一種折中 但是在MSR的處理過程中，必須將RGB三個顏色分量 獨立計算，會有可能無法重現出正常的顏色、色偏、出現光暈現象等。 MSRCR 所以為了使MSR運算後的彩色圖像，能夠重現正常的圖像顏色，需考慮原始圖像的顏色資訊，根據MSR得到的結果按比例進行調整。

Color Balance with Zone System Color cast detection RGB  YCbCr Probable adjusted pixels  CbCr Color balance Adjusted pixels (Rgain, Ggain, Bgain) 灰卡(Gray Cards) 灰卡是白平衡的依據，他是由灰色面與白色面組成，灰色面具有反射18%光線的能力，一般拿來作為測光使用，白色面具有反射90%光線的能力，可拿來作手動白平衡的標準。 修正色偏的策略 若在相片的亮部或灰色的中性區域發現色偏，大概可以判定這種色偏是全面性的，亦即暗部也有相同的色偏問題。對於這種全面性的色偏，我們會分成「亮部」、「中間調」、「暗部」三個區域分別修正，通常先修正「亮部」和「中間調」部分的色偏，因為這兩塊區域的問題會比較明顯，就大部分的情況而言，只要修除了這兩塊區域的色偏，暗部的色偏通常也會一併解決。

YCbCr YCbCr Video standards and digital photography Y (luminance), Cb(blue-difference), Cr(red-difference)

YCbCr v.s. Color Cast

Zone System Ansel Adams & Fred Archer, in 1940 I Ⅱ Ⅲ Ⅳ Ⅴ Ⅵ Ⅶ Ⅷ Ⅸ

Concept of the Proposed Method 0.5 ZoneVlower 2 ZoneVupper ZoneS

Color Cast Detection

Color Balance

Experimental Results

Subjective Evaluation camera GW camera GW Proposed MSR Proposed MSR

CIELAB Delta E Color difference (or distance) metric The International Commission on Illumination (CIE) in 1976 The small value of indicates the small distance between two color images

Objective Evaluation

Luminance & Contrast Compensation

Key Tone Low Key Low Key ↓ Middle Key

Contrast High Contrast Low Contrast Contrast  Standard deviation

Luminance & Contrast Compensation Algorithms

Existed Algorithms Log Transform Gamma Transform Histogram Equalization Global Contrast Enhancement for Histogram Modification (GCE-HM) (Arici et al. in 2009) Human Visual System Based Image Enhancement (HVSBIE) (Panetta et al. in 2008) Curvelet Transform (Starck et al. in 2003) Color Compensation algorithm using Mean Shift and Sigma Filter (ICCMS) (Han et al. in 2009)

Level-base Compounded Logarithmic Curve Function (LCLCF)

Level-base Compounded Logarithmic Curve Function

RGB v.s. HSV B R G H V S

The Characterization of Pixels Gaussian filter Low pass filter blurred image Background Showing dark and light areas Reference intensity level Original intensity level Characteristic intensity level

Reference Intensity Level the th pixel in the reference intensity level Mi Ri

Characteristic Intensity Level The th pixel in the characteristic intensity level The th pixel in the adjusted characteristic intensity level 加參數調整

Target Intensity Level I. C-type compounded log curve function in HSV 加參數調整

Experimental Results Low key image using C-type in HSV 1 2 4 3 original street image enhanced image (3) and (4) are reference intensity level before and after blending with original intensity level 4 3

Experimental Results Using C-type in HSV → n value

Experimental Results Using C-type in HSV → t value

Performance Comparison Using C-type in HSV Original image CLCLF (C-type)

Performance Comparison Using C-type in HSV Original image Log

Performance Comparison Using C-type in HSV Original image Gamma

Performance Comparison Using C-type in HSV Original image HE

Performance Comparison Using C-type in HSV Original image GCE-HM

Performance Comparison Using C-type in HSV Original image HVSBIE

Performance Comparison Using C-type in HSV Original image Curvelet

Performance Comparison Using C-type in HSV Original image ICCMS

UQI Universal Quality Index (UQI) To model any image distortion as a combination of three factors: loss of correlation, luminance distortion, and contrast distortion. The dynamic range of UQI is [-1, 1] and the larger value means the better image quality.

SSIM Structural Similarity Index (SSIM) To assess the qualitative visual appearance and similarity of luminance, contrast, and structure between two images. The dynamic range of SSIM is [0, 1] and the larger value indicates the better image quality.

Performance Comparison Using C-type in HSV The average of the result using a sliding filter (overlay) sized as 20x20 which returns the local standard deviation. The Average of Standard Deviations (ASTDs) could be an easy benchmark to measure the contrast. Though it may mislead when image contains strong noises, but in this case, enhancing a slightly noised image, it should work fine.

Target Intensity Level II. S-type compounded log curve function in RGB 加參數調整

Experimental Results Using S-type in RGB → Saturation restoration → β value Adding exposure by Photoshop CS5 Original image ERA-1 ( ) ERA-2 ( )

Experimental Results Using S-type in RGB → Luminance restoration → n2 value Adding exposure by Photoshop CS5 Original image ERA-3 ( ) ERA-4 ( )

Experimental Results Using S-type in RGB → Comparing original image with restored image

Experimental Results Using S-type in RGB → Comparing original image with restored image by CIELAB Delta E

Experimental Results Low contrast using second strategy Original Image with Thin Fog Reference Intensity Level Degraded Turbid Image Reference Intensity Level Blending with Original Intensity Level

Experimental Results Different tones  n1, n2 value

Experimental Results Different Brightness  tk value

Performance Compaison Color Restoration Original Image LCLCF with S-type in RGB LCLCF with C-type in HSV

Performance Compaison Color Restoration Gamma MSR MSRCR

Performance Comparison Color Restoration

Performance Comparison Color Balance  Removed color cast Original Image LCLCF LCLCF LCLCF

Performance Comparison Color Balance Original Image MSR MSRCR LCLCF

Performance Comparison Original Turbid Image LCLCF MSR MSRCR

Performance Comparison Original Turbid Image LCLCF MSR MSRCR

References Jzau-Sheng Lin, Shen-Chuan Tai, and Yu-Yi Liao “Color Correction with Zone System for Color Image,” International Journal of Digital Content Technology and its Applications, vol. 6, no. 10, pp. 257-265, June, 2012. (EI). Yu-Yi Liao, Jzau-Sheng Lin, Ping-Jui Liu and Shen-Chuan Tai, “Automatic Contrast enhancement Using Pixel-based Calibrating and Mean Shift Clustering,” Lecture Notes in Electrical Engineering, No. 1, vol. 128, pp. 485-489, 2012 (EI). Yu-Yi Liao, Shen-Chuan Tai, Jzau-Sheng Lin*, and Ping-Jui Liu, “Degradation of turbid images based on the adaptive logarithmic algorithm,” Computers and Mathematics with Applications, vol. 64, pp. 1259-1269, 2012. (SCI) Jzau-Sheng Lin, Ping-Jui Liu, Yu-Yi Liao, and Shen-Chuan Tai, “'Level-base Compounded Logarithmic Curve Function for Color Image Enhancement ,” IET Image Processing, (Accepted, SCI).

Thank You !