Image Processing : Basic Concept

Imaging Systems Overview Consists of two primary components: Hardware – Image acquisition system, computer, and display devices Software – Image manipulation(操作), analysis, and processing

Image is accessed (存取) as a 2-D array (陣列) of data, where each data point is referred to as a pixel (像素) Notation: I(r,c) = Brightness (亮度) of image at the pt (r,c) where r = row(列), and c = column(行)

Visible Light Imaging which objects (物體) reflect light Reflectance (反射) function determines manner in which objects (物體) reflect light

Sensors : Converts (轉換) light energy into electrical energy a) Single imaging sensor ; b) Linear ( line) sensor ; c) 2-D or array sensor CCD: 4kx4k CMOS: less power, cheaper, image quality not as good as CCD

Image Representation Optical (光學) image : Collection of spatially distributed (空間分佈) light energy measured by an image sensor to generate I(r,c) Matrix: 2-D array like the image model, I(r,c) Vector: One row or column in a matrix

Image Types Binary (二元化) images: Simplest type of images, which can take two values, typically black or white, or “0” or “1” Gray scale (灰階) images: One-color or monochrome images that contains only brightness information and no color information Color images: 3 band monochrome images, where each band corresponds to a different color, typically red, blue and green or RGB

Color pixel vector: Single pixel’s values for a color image, (R,G,B) Multispectral(多光譜) Images: Images of many bands containing information outside of the visible spectrum(可見光譜)

Color Transform/Color Model Mathematical model or algorithm to map(映射) RGB data into another color space (色彩空間) Decouples (分開) brightness and color information Hue(色調)/Saturation(飽和度)/Lightness(亮度) (HSL) Color Transform: Describes colors in terms that we can more readily understand

Hue corresponds to color, saturation corresponds to the amount of white in color, and lightness is the brightness For example: a deep, bright orange color would have a large intensity (bright), a hue of “orange” , and a high value of saturation (“deep(深度)”) But in terms of RGB components, this color would have the values as R =245, G= 110, and B=20

Equations for mapping RGB to HSL are: where

Digital Image File Formats Bitmap images (raster images): Images that can be represented by our image model, I (r,c)

Image file header (檔案標頭): A set of parameters (參數) found at the start of the file image and contains information regarding: Number of rows (列數)(height, 高) Number of columns (行數)(width, 寬) Number of bands (頻帶數) Number of bits per pixel (每個像數之位 元數)(bpp) File type (檔案形式)

Look-up table (LUT): Used for storing RGB values for 8-bit color images

Common image file formats are: BIN, RAW PPM,PBM,PGM BMP JPEG TIFF GIF RAS SGI PNG PICT, FPX EPS VIP

Matlab 影像處理I/O相關指令簡介 在Matlab中，每一個像素(pixel)之顏色係以0到1之數值來表示。1代表最亮，0代表最暗。 彩色的組成係由RGB，也就是 red(紅) 、green(綠) 、blue(籃)的色階來構成。 影像之處理係基於矩陣的運算，矩陣內的數值，代表了影像的強弱。

Matlab 影像處理I/O相關指令簡介 黑白影像用二維的矩陣來表示，彩色的影像則需藉助三維的矩陣來表示RGB 在該一位置的色濃度。

Matlab 影像處理I/O相關指令簡介 Show 圖指令imshow( ) 其指令格式為：imshow(影像矩陣 A,色階 N) ，代表影像矩陣 A以N個色階來顯示黑白影像。 若N未設定，則對24位元之影像，內定為256色階。 影像矩陣A以三維空間A( : , : , 3)來表示彩色影像，A(: , : , 1)表示紅色之色階； A(: , : , 2)表示綠色之色階； A(: , : , 3)表示籃色之色階。

Matlab 影像處理I/O相關指令簡介 另外一種設定為設定界限值，以便讓圖形在範圍以下顯示黑色，在範圍以上顯示白色。指令格式為：imshow(影像矩陣 A, [ lim_l lim_h]) 代表影像矩陣 A 中的值若小於lim_l，則顯示黑色；若大於lim_h，則顯示白色。

Matlab 影像處理I/O相關指令簡介 若已經有了現成的影像檔，希望讀入Matlab的workspace中，可用imread(‘檔案名稱’)讀入，再用imshow( )顯示出來即可。 imwrite( )可用來將影像寫入影像檔中，其指令格式為：imwrite(影像來源, 寫入之影像檔名稱, ‘影像格式’)

Ex2_1.m clear; close all; A=imread('1.bmp'); figure % 開繪圖視窗 imshow(A); size(A) figure imshow(A(:,:,2)); %僅show綠色部份 imwrite(A(:,:,2),'ex2_1.tif','tif'); B=A(100:150,150:200,1); %取紅色之部份影像 imshow(B); imshow(B,[100 200]);

Convolution Mask (迴旋遮罩) Mask Operation (遮罩運算) 遮罩通常為 3 x 3 (有時使用 5 x 5, 7 x 7)

Convolution: Consist of following process Overlay (覆蓋) the mask (遮罩) on the image Multiply the coincident terms (乘上一致的項次) Sum all the results (將結果加總) Move to the next pixel, across the entire image Convolution mask for first order hold (一階保持之迴旋遮罩)

Spatial Filters (空間濾波器) Operate on raw (原始的) image data in the (r,c) space, by considering small neighborhoods (鄰居), 3x3, 5x5, 7x7, and moving sequentially (依序地)across and down the image Returns a result based on a linear (線性) or nonlinear (非線性) operation

Consists of three types of filters: Mean filters (均值濾波器) Median filters (中值濾波器) Enhancement filters (增強濾波器) Many spatial filters are linear filters implemented with a convolution mask (以迴旋遮罩來實現); the result is a weighted sum (加權和) of a pixel and its neighbors

Mask coefficients (係數) tend to effect (影響) the image in the following general ways: Coefficients are positive (正數): blurs (模糊化) the image Coefficients are alternating positive and negative (正負交替): sharpens (銳化) the image Coefficients sum to 1 (係數總和為1): brightness retained (保留) Coefficients sum to 0 (係數總和為1): dark (暗) image

Mean filters (均值濾波器) : Averaging filters (平均濾波器) Tend to blur (模糊化) the image Adds a softer look to the image Example 3x3 convolution mask (迴旋遮罩):

Mean filter Original image Mean filtered image

Median filters (中值濾波器) : Nonlinear filter Sorts (排序) the pixel values in a small neighborhood and replaces the center (中央的) pixel with the middle value (中間值) in the sorted list (已排序串列) Output image (輸出影像) needs to be written to a separate image (a buffer (緩衝器)), so that results are not corrupted (破壞) Neighborhood (鄰居) can be of any size but 3x3, 5x5 and 7x7 are typical (典型)

salt and pepper noise (胡椒鹽雜訊) Median filter Original image with salt and pepper noise (胡椒鹽雜訊) Median filtered image (3x3)

Ex3_2.m (mean filter 與 median filter之練習) %Image: 原影像 %Image_noisy: 受污染的影像% %Image_low: 執行低通濾波器後的影像% %Image_med: 執行中值濾波器後的影像% Image=imread('ex2_1.tif'); %讀取影像 %將胡椒鹽式的雜訊加入原影像中，其中0.06 是雜訊密度% Image_noisy= imnoise (Image ,'salt & pepper',0.06); Image2_noisy=double(Image_noisy)/255; %變為 double格式 h=[1/9 1/9 1/9;1/9 1/9 1/9 ; 1/9 1/9 1/9]; %濾波器的子遮罩 Image_low=filter2(h, Image2_noisy); %執行低通濾波 Image_med=medfilt2(Image_noisy,[3 3]); %中值濾波採3×3矩陣 imshow(Image) figure,imshow(Image_noisy) figure,imshow(Image_low) figure,imshow(Image_med)

Enhancement filters (增強濾波器) : Implemented with convolution masks having alternating positive and negative (正負交替) coefficients Enhance the image by sharpening (銳化) Two types considered here: Laplacian-type (拉普拉斯型) filters Difference filters (差分型)

1. Laplacian-type (拉普拉斯型) filters : Are rotationally invariant (與旋轉無關), that is they enhance the details (增強細節) in all directions equally (所有方向皆相等) Example convolution masks of Laplacian-type filters are: Filter 1 Filter 2 Filter 3

Laplacian filter Original image Laplacian filtered image Contrast enhanced (對比增強) Version of Laplacian filtered image

2. Difference (差分) filters: Also called as emboss (浮雕) filters Enhances the details in the direction specific (特定方向) to the mask selected Four primary difference filter convolution masks, corresponding to the edges (邊) in the vertical (垂直), horizontal (水平), and two diagonal directions (對角方向) are: Vertical Horizontal Diagonal 1 Diagonal 2

Difference filter (對比增強) Original image Difference filtered image added to the original image, with contrast enhanced (對比增強)

Ex3_3.m (增強濾波器之練習) close all; clear; A=imread('cat.bmp'); [m,n]=size(A); %原影像之size為mxn imshow(A); CM_lapa=[0 -1 0; -1 5 -1; 0 -1 0]; %Laplacian迴旋遮罩 CM_diff=[1 0 0; 0 1 0; 0 0 -1]; % difference 迴旋遮罩 B=filter2(CM_lapa,A); %執行Laplacian 遮罩運算 C=filter2(CM_diff,A); %執行difference 遮罩運算 figure; imshow(B/256); figure; imshow(C/256);

Binary Image Analysis (二元影像分析) Binary images are useful in many computer vision applications which require simple object shape; such as positioning a robot to grasp (抓取) an object, to check a manufactured object for defects (瑕疵), FAX, OCR(光學字元辨識)

Binary Image Analysis (二元影像分析) Most cameras provide us color or gray level images, thus we need to convert those images into binary images Next, we extract (擷取) simple binary features and use them to classify (分辨) binary objects

Thresholding via Histogram (經由直方圖做臨界化) Thresholding is required to create a binary image (二元影像) from a gray level image (灰階影像) This is done by specifying a threshold value (指定一臨界值) which will set all values above the specified gray level to ‘1’ and everything below the specified value to ‘0’ Typically 255 is used for ‘1’ and 0 is used for the ‘0’ value

In many applications the threshold value is determined experimentally (由實驗求得) and is highly dependent on lighting conditions and object to background contrast (與光線條件,目標物與背景之對比度有密切關係) It is much easier to find a good threshold value with proper lighting, and good background to object contrast

Figure 3.3-1: Effects of Lighting and Object to Background Contrast on Thresholding a) An image of a bowl with high object to background contrast (目標物與背景之 對比度高) and good lighting (照明良好) b) Result of thresholding (臨界化) image (a)

Figure 3.3-1: Effects of Lighting and Object to Background Contrast on Thresholding (contd) c) An image of a bowl with poor object to background contrast (目標物與背景之 對比度低) and poor lighting(照明不佳) d) Result of thresholding image (c)

The histogram (直方圖) is a plot of gray level versus the number of pixels (灰階對像素個數之作圖) in the image at each gray level Histogram of an image is examined (檢視) to select the proper (適當的) threshold value The peaks (尖峰) and valleys (深谷) in the histogram are examined and a threshold is experimentally selected (經由實驗選取) that will best separate (最佳的分割) the object from the background (背景)

Figure 3.3-2: Histograms (直方圖) Threshold  b) The histogram of image (a), showing the threshold that separates object and background (一能將目標物與背景 分開之臨界點) a) An image of a bowl with high object to background contrast and good lighting Image after threshold (經臨界化後之影像)

Figure 3.3-2: Histograms (contd) Threshold  d) The histogram of image (c), showing what appears to be a good threshold, but it does not successfully separate object and background c) An image of a bowl with poor object to background contrast and poor lighting Image after threshold

Ex3_5.m (臨界化之練習) A=imread('car_number.bmp'); imshow(A); X=A(:,:,3); [m,n]=size(X); figure, imshow(X); figure, imhist(X); %繪出此影像之直方圖 th=70; %設定臨界值 C=zeros(m,n); for i=1:m for j=1:n if X(i,j)>th; C(i,j)=1; end; figure,imshow(C);

Histogram equalization (柱狀圖等化) A technique where the histogram of the resultant image is as flat as possible (產生一柱狀圖很平之影像) The theoretical basis for histogram equalization involves probability theory, where we treat the histogram as the probability distribution of the gray levels (將柱狀圖看成灰階之機率分怖) Its function is similar to that of a histogram stretch but often provides more visually pleasing results across a wider range of images (與柱狀圖伸展類似但常能產生視覺效果更佳之影像)

Consists of 4 steps: 1. Find the running sum of the histogram values (求出柱狀圖值之移動和) 2. Normalize the values from step (1) by dividing by the total number of pixels (將(1)所 得之值除以像素之總數) 3. Multiply the values from step (2) by the maximum gray level value and round (將(2)所 得之值乘以最大灰階值並捨去小數部份) 4. Map the gray level values to the results from step (3) using a one-to-one correspondence (將灰階值以一對一對應之方 式映射至步驟(3)所得之結果)

Example: Given the following histogram: Number of Pixels 3-bits per pixel image – range is 0 to 7. Given the following histogram: Number of Pixels Gray Level Value (Histogram values) 0 10 1 8 2 9 3 2 4 14 5 1 6 5 7 2

1) Create a running sum of the histogram values 1) Create a running sum of the histogram values. (求出柱狀圖值之移動和) This means the first value is 10, the second is 10+8=18, next 10+8+9=27, and so on. Here we get 10, 18, 27, 29, 43, 44, 49, 51 2) Normalize by dividing by the total number of pixels. (將(1)所 得之值除以像素之總數) The total number of pixels is: 10+8+9+2+14+1+5+0 = 51 (note this is the last number from step 1), so we get: 10/51, 18/51, 27/51, 29/51, 43/51, 44/51, 49/51, 51/51 3) Multiply these values by the maximum gray level values, in this case 7, and then round the result to the closest integer (將(2)所得之值乘以最大灰階值並捨去小數部份). After this is done we obtain: 1, 2, 4, 4, 6, 6, 7, 7

4) Map the original values to the results from step 3 by a one-to-one correspondence (將灰階值以一對一對應之方式映射至步驟(3)所得之結果). This is done as follows: Original Gray Histogram Level Value Equalized Values 0 1 1 2 2 4 3 4 4 6 5 6 6 7 7 7

All pixels in the original image with gray level 0 are set to 1, values of 1 are set to 2, 2 set to 4, 3 set to 4, and so on. After the histogram equalization values are calculated and can be implemented efficiently with a look-up-table (LUT)(柱狀圖等化值算出後，可以查表法實現), as discussed in Chapter 2 We can see the original histogram and the resulting histogram equalized histogram in Fig. 8.2.14. Although the result is not flat, it is closer to being flat than the original histogram(雖然結果並非平坦，它比原本之柱狀圖要平坦多了)

Histogram Equalization Examples (柱狀圖等化範例) 1. Input image Resultant image after histogram equalization

Histogram Equalization Examples (contd) 2. Input image Resultant image after histogram equalization Note: As can be seen histogram equalization provides similar results regardless of the input image (可看出不管輸入影像之亮度如何， 柱狀圖等化提供類似的結果)

Histogram equalization of a digital image will not typically provide a histogram that is perfectly flat, but it will make it as flat as possible (通常並不一定產生一平坦之柱狀圖，但卻會使柱狀圖儘可能的平坦) Histogram equalization may not always provide the desired effect, since its goal is fixed – to distribute the gray level values as evenly as possible. (將灰階值儘可能地平均分佈)To allow for interactive histogram manipulation, the ability to specify the histogram is necessary

Ex8_3.m (柱狀圖等化法) %柱狀圖等化法% %image_1 原影像；image_2 執行柱狀圖等化後的影像 image_1=imread('L5_1.bmp'); %讀取影像 image_2=histeq(image_1);%對上述影像執行柱狀圖等化 imshow(image_1) %顯示原影像 figure,imshow(image_2) %顯示處理後的影像 figure,imhist(image_1) %顯示原影像的柱狀圖 figure,imhist (image_2) %顯示處理後影像的柱狀圖

Morphological Filtering (形態濾波) Morphology(形態學) relates to form and structure of objects (物件之形勢與結構) Morphological filtering simplifies a segmented image to facilitate the search for objects of interest(形態濾波簡化一分割影像,有助於搜尋感興趣之物件) This is done by smoothing out object outlines (將物件之外形平滑化), filling small holes (將小洞填滿), eliminating small projections (消除小投影), and with other similar techniques

The two principal morphological operations(形態運算子) are dilation (膨脹)and erosion (侵蝕) Dilation allows objects to expand (延伸), thus potentially filling in small holes (將小洞填滿), and connecting disjoint objects (連接不相連之物件) Erosion shrinks (萎縮) objects by etching away (eroding) (侵蝕) their boundaries (邊界)

These operations can be customized for an application by the proper selection of the structuring element (正確地選取結構元素), which determines exactly how the objects will be dilated or eroded Basically, the structuring element is used to probe (探測) the image to find how it will fit, or not fit, into the image object(s)

Dilation(膨脹): It is performed by laying the structuring element on the image and sliding it across the image in a manner similar to convolution (將結構元素以類似迴旋的方式在影像中滑動) If the origin of the structuring element coincides with a ‘0’ in the image, there is no change; move to the next pixel (若結構元素之原點對應至影像中之‘0’ ,沒有改變,移至下一像素) If the origin of the structuring element coincides with a ‘1’ in the image, perform the OR logic operation on all pixels within the structuring element (若結構元素之原點對應至影像中之‘1’ ,將結構元素內之所有像素執行OR邏輯運算)

With a dilation operation, all the ‘1’ pixels in the original image will be retained (保留), any boundaries will be expanded (邊界將被延展), and small holes will be filled (小洞將被填滿)

Note: In 1st printing of book, the structuring element is incorrect

Erosion (侵蝕) : It is similar to dilation, but we turn pixels to '0', not '1' If the origin of the structuring element coincides with a '0' in the image, there is no change; move to the next pixel (若結構元素之原點對應至影像中之‘0’ ,沒有改變,移至下一像素) If the origin of the structuring element coincides with a ‘1’ in the image, and any of the ‘1’ pixels in the structuring element extend beyond the object (‘1’ pixels) in the image, then change the ‘1’ pixel in the image, whose location corresponds to the origin of the structuring element, to a ‘0‘(若結構元素之原點對應至影像中之‘1’,且結構元素內之‘1’ 超越了影像中之‘1’ ,則將結構元素之原點所對應之像素變成‘0‘).

With an erosion operation, the only remaining pixels are those that coincide to the origin of the structuring element where it is all contained in the object

Note: In first printing of book, the 4th row, 4th col ‘0’, should be a ‘1’, in the IMAGE

Opening (斷開): It consists of an erosion followed by a dilation (先侵蝕再膨脹) It can be used to eliminate all pixels in regions that are too small to contain the structuring element (可去除小到無法包含結構元素之區域內之所有像素) In this case the structuring element is often called a probe, as it is probing the image looking for small objects to filter out of the image

The output image tends to take a shape similar to the structuring element itself

Closing (閉合): It consists of a dilation followed by erosion(先膨脹再侵蝕) It can be used to fill in holes and small gaps (填補洞與空隙) It will connect small, adjacent objects (連接小且相鄰之物件) Closing tends to “close up” or “fill in” objects

Note that holes and gaps are filled, but, unlike dilation, more of the original boundary is retained

Closing and opening will have different results(斷開與閉合結果不同)even though both consist of an erosion and a dilation Therefore, order of operation is important (運算之順序很重要) for morphological operations Different structuring elements will also provide different results (不同之結構元素將產生不同之結果). As noted before, objects in the output image will tend to take the shape of the structuring element (輸出影像中之物件傾向於保有結構元素之形狀)

Figure 4.3-18: Binary Dilation with Various Shape Structuring Elements a) Original image, a microscope cell image that has undergone a threshold operation (original image courtesy of Sara Sawyer, SIUE) b) Dilation with a circular (圓形的)structuring element

Figure 4.3-18: Binary Dilation with Various Shape Structuring Elements (contd) c) Dilation with a square (方形) structuring element d) Dilation with a cross shape (十字形)structuring element

Figure 4.3-19: Dilation with Different Size Structuring Elements a) Original image b) Dilation with a circular structuring element of size 3

Figure 4.3-19: Dilation with Different Size Structuring Elements (contd) c) Dilation with a circular structuring element of size 7 d) Dilation with a circular structuring element of size 11

Figure 4.3-20: Binary Erosion with Various Shape Structuring Elements a) Original image, a microscope cell image that has undergone a threshold operation (original image courtesy of Sara Sawyer, SIUE), b) Erosion with a circular structuring element

Figure 4.3-20: Binary Erosion with Various Shape Structuring Elements (contd) c) Erosion with a square structuring element d) Erosion with a cross shape structuring element

Figure 4.3-21: Binary Opening with Various Shape Structuring Elements a) Original image, a microscope cell image that has undergone a threshold operation (original image courtesy of Sara Sawyer, SIUE), b) Opening with a circular structuring element

Figure 4.3-21: Binary Opening with Various Shape Structuring Elements (contd) c) Opening with a square structuring element d) Opening with a cross shape structuring element

Figure 4.3-22: Binary Closing with Various Shape Structuring Elements a) Original image, a microscope cell image that has undergone a threshold operation (original image courtesy of Sara Sawyer, SIUE), b) Closing with a circular structuring element

Figure 4.3-22: Binary Closing with Various Shape Structuring Elements (contd) c) Closing with a square structuring element d) Closing with a cross shape structuring element

Figure 4.3-23: Opening and Closing with Different Size Structuring Elements a) Original microscopic cell image (courtesy of Sara Sawyer, SIUE) b) Image after undergoing a threshold operation

Figure 4.3-23: Opening and Closing with Different Size Structuring Elements (contd) c) Opening with a circular structuring element of size 5 d) Closing with a circular structuring element of size 5