CMOS果冻效应

W.-h. Cho and K.-S. Hong: Affine Motion Based CMOS Distortion Analysis and CMOS Digital Image Stabilization Contributed Paper Manuscript received June 28, 2007 0098 3063/07/$20.00 © 2007 IEEE 833 Affine Motion Based CMOS Distortion Analysis and CMOS Digital Image Stabilization Won-ho Cho and Ki-Sang Hong Abstract — The CMOS image distortion is due to the rolling shutter in CMOS image sensors (CISs) and it can be more exaggerated when a CIS camera moves rapidly. Several methods have been proposed to remove CMOS distortions made by the translational motion. But, in this paper, we propose the affine motion based CMOS distortion correction method combined with digital image stabilization. To remove CMOS distortions due to the affine global image motion, CMOS distortion model is proposed to explain the effect of the affine global image motion on the CMOS distortion in CISs. To improve CMOS video's visuals, we propose CMOS digital image stabilization to remove the jittering motions in the new image sequence obtained by our correction method. In addition, to reduce the computational time and the outlier effect, a reliable feature selection method is proposed to be used in the affine global image motion estimation. In the experiment results, we show that the proposed CMOS distortion correction method is more general than previous ones. Also, we show that our stabilization method can improve CMOS video's visuals running in real-time1. Index Terms — CMOS Image Sensor, Affine Global Image Motion, CMOS Distortion Model, Rolling Shutter, Digital Image Stabilization. I. INTRODUCTION Recently, the imaging device market of CMOS imaging sensors (CISs) has been remarkably expanding due to the increasing demand for three key applications: mobile phones, digital cameras, and digital camcorders. For a long time, CCDs as a leading image sensor technology has been dominating high-end digital imaging products. But, owing to the advance of semiconductor technology and the increasing popularity of the key applications, CISs are now quickly replacing CCDs in these applications. Even in the high-end imaging products such as D-SLR cameras, CISs are now gaining an advantage unseating CCDs as a leading image sensor. However, the images captured by CISs can have CMOS image distortions due to the shuttering mechanism, which is more visible in low-end CIS products like PC webcams or mobile-phones. CISs, unlike CCDs, have a rolling shutter. It reads out the scanline one by one in the sensor while other several scanlines are exposed except for the current read-out scanline. Thus, if either a CIS camera moves 1 Won-ho Cho is with Division of Electrical and Computer Engineering, POSTECH, Pohang, Gyungbuk, S. Korea (e-mail: ellescho@postech.ac.kr). Ki-Sang Hong is with Division of Electrical and Computer Engineering, POSTECH, Pohang, Gyungbuk, S. Korea (e-mail: hongks@postech.ac.kr). or objects in the scene are not stationary, the rolling shutter mechanism will cause CMOS distortions in CMOS images. Fig. 1. CMOS distortions caused by three movements of a CIS PC webcam: (a) a standstill, (b),(c) the rotational motions, (d) the upward motion, (e) the right-directional motion. Fig. 1 shows examples of five CMOS images taken by a CIS PC webcam. Fig. 1(a) was captured when the camera was stationary and the others were captured when it was in motion. Fig. 1(b) and (c) show distorted CMOS images due to the rotational motions of the CIS camera which make vertical lines of the scene appear curved. Fig. 1(d) shows the CMOS distortion due to the upward motion which makes the scene objects appear vertically squeezed compared to Fig. 1(a). Also, the right-directional motion of the camera in Fig. 1(e) makes the same vertical lines slant to the left. Because these CMOS distortions degrade the image quality, researchers are now interested in the CMOS distortion [1]. Especially, since the CMOS distortion is associated with the global image motion between two consecutive frames, several works have tried to clarify the relationship between the CMOS distortion and the global image motion [2]-[4]. The concept of CMOS distortion was first suggested by Hwang [2]. The author solved the CMOS distortion problem by making a distortion transformation based on the translational motion, but it was restricted to the translational motion. Similarly, Liang et al. [3] used the translational motion to remove CMOS distortions and the CMOS distortion correction algorithm was based on the rolling shutter mechanism. However, it is difficult to use an affine global image motion in such algorithm because their method corrects CMOS distortions by shifting the position of each scanline of a CMOS image. To solve the CMOS distortion problem more generally, Im et al. [4] made the CMOS distortion Authorized licensed use limited to: UNIVERSITY OF INCHEON. Downloaded on June 2, 2009 at 01:41 from IEEE Xplore. Restrictions apply. IEEE Transactions on Consumer Electronics, Vol. 53, No. 3, AUGUST 2007 834 transformation with the translational motion under the parameterized homography framework. However, it is not easy to use the more complex motion model in their framework. The global image motion affects the overall CMOS video's visuals as well as the CMOS distortion of each CMOS image if the motion is not smooth. Thus, to improve CMOS video's visuals, CMOS video sequences need to be made without jittering motions [4]. Fig. 2. The block diagram of our CMOS DIS algorithm. Fig. 2. shows the basic block diagram of our CMOS digital stabilization (CMOS DIS) algorithm. Here, the CMOS distortions in CMOS images are removed through CMOS Distortion Correction using the global image motion from Global Motion Estimation. The global image motion is important in our CMOS DIS algorithm and it can be estimated by using the block matching algorithms (BMAs) [5]-[8] or the optical flow methods [9]-[10]. BMAs as fast 2D local motion estimators are generally used in video encoding techniques [11]-[12]. But, since the affine motion can be estimated with at least three matching pairs between two consecutive images, it is important to select reliable features which give the best matching results [5]. In this paper, we present the affine motion based CMOS distortion correction method and a real-time CMOS digital image stabilization algorithm. The proposed CMOS distortion correction method can provide a general solution to handle the more complex CMOS distortions in CMOS images. Also, the proposed CMOS DIS can make CMOS video’s visuals more stable by stabilizing the new CMOS video obtained by our CMOS distortion correction method. The remainder of this paper is organized as follows: Section II describes the proposed CMOS distortion model to analyze CMOS distortions based on the affine global image motion in a CIS. Section III explains the fast and robust method to compute the affine motion parameters with reliable feature matching pairs even in the case that a CMOS video contains moving objects. Section IV shows the overall procedure of the proposed CMOS DIS algorithm combined with our CMOS distortion correction method. In Section V, we analyze the results from our CMOS distortion correction method under several motions of a camera showing that it improves CMOS video’s visuals. Finally, conclusions are drawn in Section VI. II. CMOS DISTORTION ANALYSIS This section describes the proposed CMOS distortion model based on the affine global image motion. The CMOS distortion model can explain the effect of the global image motion on the CMOS distortion in a CIS. From this model, we can derive the mapping function between a CMOS image and its undistorted one and the function is used to remove CMOS distortions in our CMOS distortion correction method. Fig. 3 shows how the CMOS distortion is related to the affine global image motion by the rolling shutter. (a) (b) Fig. 3. The rolling shutter mechanism and CMOS distortion in CISs: (a) the affine transformation defined over two consecutive CMOS frames, (b) the illustration of CMOS distortion in a CMOS image. In Fig. 3(a), two consecutive CMOS frames 1I c n− and I c n containing a distorted rectangular object can be related by 1 1 1 11 1 1 10 0 1 n n nc c c c n nAn n n n n n n n a b e c d f − − − ⎛ ⎞⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞⎛ ⎞⎜ ⎟= = =⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟⎜ ⎟⎜ ⎟ ⎝ ⎠⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠⎜ ⎟⎝ ⎠ A bx x x x T 0 , (1) where 1 c n−x and c nx represent the matching positions between 1I c n− and I c n , and A nT is the 3×3 affine transformation. Fig. 3(b) illustrates how a rectangular scene object is distorted by the rolling shutter. At the beginning of scan, 0t = , the scene object virtually has an undistorted rectangular shape as drawn by the dashed line in Fig. 3(b) and its feature point at the undistorted position ( ),i i ix y=x is on the object. Here, we assume two things for our CMOS distortion model. First, the undistorted scene object in Fig. 3(b) moves by AnT during the one frame time sT because the camera motion is given by A nT in Fig. 3(a). Second, the feature point on the scene object in Fig. 3(b) moves with uniform velocity ( )iv x during sT because sT is too short in CISs. Thus, by the two assumptions, ( )iv x is ( )An i sTu x and ( )An iu x is given by ( ) ( 1) ( 1) n i n i nA n i n i n i n i n i n a x b y e c x d y f − + +⎛ ⎞= + − = ⎜ ⎟+ − +⎝ ⎠ u x A x b x , (2) where nA and nb form A nT as in (1). Authorized licensed use limited to: UNIVERSITY OF INCHEON. Downloaded on June 2, 2009 at 01:41 from IEEE Xplore. Restrictions apply. W.-h. Cho and K.-S. Hong: Affine Motion Based CMOS Distortion Analysis and CMOS Digital Image Stabilization 835 A nT is the transformation between 1I c n− and I c n . But, by the assumptions, we can set the displacement at ix to ( )An iu x during sT and thus, the uniform velocity ( )iv x is obtained by dividing ( )An iu x by sT . After some time, xt t= which is the elapsed scanning time to the distorted position cix starting from the image origin, the feature point at ix is exposed at ( ),c c ci i ix y=x satisfying the following equation, ( ) ( ) 0 0 1x xt tc A i i i i n it t s dt dt T= = = + = +∫ ∫x x v x x u x , (3) where cx i st y T H≈ ⋅ and H is the number of scanlines. This is the proposed CMOS distortion model based on the affine global image motion. It means that cix is distant from ix by the displacement ( )0xt An i st T dt=∫ u x which makes a CMOS image appear distorted. Since we assume that ( )iv x is the uniform velocity for sT , (3) can be written by ( ) cc A ii i n i yH= +x x u x . (4) By substituting (2) into (4) and then rearranging (4), we can derive the CMOS distortion function ( )nD x between the CMOS image Icn and its undistorted In , which is given by ( ) ( )( ) ( )( ) ( )( ) 1 1 1 c n n n n n n n n n n a x b y e y x H c x d y f Hy H c x d y f = ⎛ ⎞− + ++⎜ ⎟− + − +⎜ ⎟= ⎜ ⎟⎜ ⎟⎜ ⎟− + − +⎝ ⎠ x D x . (5) Its inverse function 1( )cn −D x has the form of ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 1 2 2 2 c n d c c c n n n n a d c n n n n a c c c c n n n n a d c n n n n K Hx e y b H f y K K b c y K H f y c Hx e y y K K b c y −= ⎛ ⎞− − −⎜ ⎟⎜ ⎟−⎜ ⎟= ⎜ ⎟− − −⎜ ⎟⎜ ⎟−⎝ ⎠ x D x , (6) where ( )1a cn nK H a y= + − and ( )1d cn nK H d y= + − . The CMOS distortion function ( )nD x represents the mapping relationship between Icn and In , which can simulate CMOS distortions observed in Fig. 1. (a) (b) (c) Fig. 4. The mapping configurations by the CMOS distortion function for three different cases of a camera motion: (a) the left translational motion, (b) the 10° counterclockwise rotational motion, (c) the 20% zoom-in motion. Fig. 4. simulates the CMOS distortions for three different cases of an affine global image motion. Here, the dashed line area (a rectangle region) with the image size 320×240 is a CMOS image Icn and its undistorted image In mapped by (6) is drawn by a solid line. In our CMOS distortion correction method, the way of removing CMOS distortions is to warp the solid line region onto the dashed line region (an original CMOS image) by using (5). For example, because the left translational motion slants the scene of a CMOS image to the left, its undistorted image has to slant to the right to remove the left-slanted distortion as in Fig. 4(a). Also, in the case of rotational motion, since the scene in Fig. 1(b) appears curved to the right, its undistorted image should be curved to the left as in Fig. 4(b). As for the 20% zoom-in case in Fig. 4(c), we can give the same reasoning to remove its CMOS distortion. But, the change of 20% in the zoom scale ( na and nd in (1)) is quite large and it does not usually occur in a real situation. III. GLOBAL MOTION ESTIMATION In this section, we will explain a fast and robust method to estimate the affine global image motion between two consecutive CMOS frames. As explained in Section II, the undistorted image can be obtained by using ( )nD x once the affine global motion is estimated from a CMOS frame sequence. To compute the motion parameters in (1), we need the set of matching features between two consecutive CMOS frames. For this, we use the matching blocks which are obtained from the block matching algorithm [12] based on the multi-resolution framework [11]. When computing the motion parameters robustly, we use the uncertainty information of useful blocks. The useful blocks are the ones which give good matching results and are selected before the block matching process. Also, to reduce the moving object effect on the motion parameters, we need a fast outlier removal method to exclude the block pairs belonging to moving objects. A. The reliable block selection for a fast motion estimation The block matching algorithm is used as a fast local motion estimation method and its local motions are usually used to compute the global image motion in the digital image stabilization [5]-[6]. However, some matching pairs do not comply with a global image motion, which can cause a Authorized licensed use limited to: UNIVERSITY OF INCHEON. Downloaded on June 2, 2009 at 01:41 from IEEE Xplore. Restrictions apply. IEEE Transactions on Consumer Electronics, Vol. 53, No. 3, AUGUST 2007 836 significant error in the process of the global motion computation. Thus, to improve the reliability and the speed of the local motion estimation by the BMA, we have to select useful blocks which have strong line or corner features within their block patches before the block matching process begins. The useful ones can be selected by using the 2×2 cornerness matrix mC [13]. The matrix has the following form of 2 2 I I I I I I x x y m y x y ⎛ ⎞< > < >= ⎜ ⎟⎜ ⎟< > < >⎝ ⎠ C , (7) where Ix and I y are the x and y-directional image gradients and < ⋅ > means the sampled mean within the m-th block patch. To select the blocks, we test whether both two eigenvalues of mC , 1λ and 2λ , are over the predefined threshold value or not [13]. This block-selection can reduce the number of blocks to be used in the block matching process depending on the predefined threshold value. After the block matching process, we use a fast outlier removal method to reduce the effect of moving objects. This method, unlike other sophisticated methods [14]-[15], can select the inlier blocks by using a simple computation. In our work, the affine global image motion AnT can be correctly estimated by using those useful matching blocks if there are no moving objects. But, if a large portion of the matching blocks belongs to moving objects, then the estimated global motion is inclined to follow the wrong motion of moving objects. In this case, our strategy is to select the matching block pairs whose local motions are close to the motions computed by the previous global motion. This strategy has the following assumptions. First, since the global motion will not change abruptly frame by frame in a real video sequence, the local motions of inlier blocks are usually close to the ones computed by the previous global motion. In addition, since the motions of moving objects are usually different from the motion of a camera, we can make a group of the inliers which are different from the one of outliers. Based on these assumptions, a method of removing the outlier blocks can be made with the following procedure. For all M matching block pairs { } 1, Mm m m=′x x between 1I c n− and I c n , we obtain the local motions ( )BMA mu x estimated by the BMA and ( )1An m−u x computed by the previous affine global motion 1 A n−T as follows: ( ) ( )1 1 1 BMA m m m A n m n m n m− − − ′= − = + − u x x x u x A x b x . (8) Using these local motions, we make a histogram of magnitude of difference 1 BMA A n−−u u and choose a lower 50% of all matching pairs along bins of the histogram, which is indicated by Inlier bins in Fig. 5 (a). Here, Bin in Fig. 5(a) means the magnitudes of 1 BMA A n−−u u and its size is a one- pixel. (a) (b) Fig. 5. A fast outlier removal method: (a) the histogram of magnitudes of differences overlaid with its cumulative histogram, (b) the BMA matching results between two consecutive CMOS images. In Fig. 5 (b), there are two CMOS images: the left is 1I c n− and the right is Icn . The inlier blocks mostly on the background were drawn by green and the outlier ones mostly on the moving object were drawn by red. In this method, the inlier region with the lower 50% comes from the following reasoning. By the first assumption, the inlier blocks usually form a cluster in the lower bins of the histogram as shown in Fig. 5(a). Also, by the second assumption, the outlier blocks form a group distant from the one of inliers in the histogram. Thus, we experimentally set the inlier region to 50% along the lower bins indicated in Fig. 5(a) without the loss of the accuracy of the motion parameters and the computational burden. B. Affine global image motion computation Once we obtain the set of reliable matching block pairs { } 1, Mm m m=′x x between two consecutive CMOS frames, the affine motion parameters in (1) can be computed by using the least square method with as follows: 1 0 0 0 0 0 0 1 T m m m T Tm T m m m m x x y y x y ′ ′ ⎛ ⎞⎛ ⎞ ⎛ ⎞= = ⎜ ⎟⎜ ⎟ ⎜ ⎟′ ′⎝ ⎠ ⎝ ⎠ ⎝ ⎠ p q Α Α , (9) where ( , )m m mx y=x , ( , )m m mx y′ ′ ′=x , the affine parameter set ( ), , , , ,n n n n n na b e c d f=A , and the frame indices are omitted for convenience in mx and m′x . But, since all the matching pairs are evenly weighted in (9), this approach can cause serious computational errors in the parameter computation. Authorized licensed use limited to: UNIVERSITY OF INCHEON. Downloaded on June 2, 2009 at 01:41 from IEEE Xplore. Restrictions apply. W.-h. Cho and K.-S. Hong: Affine Motion Based CMOS Distortion Analysis and CMOS Digital Image Stabilization 837 Fig. 6. Examples of block matching pairs: (A,A`), (B,B`), and (C,C`) are examples of matching pairs along the horizontal or vertical lines between two consecutive CMOS frames (a) and (b), and their local motions are drawn by the solid lines in (a). For example as in Fig. 6, the block matching (A,A`) and (C, C`) are located on vertical lines, and (B,B`) are on the horizontal line. The local motions of these pairs are different from their true ones computed by AnT in (1).