Facial expression recognition based on 3D face reconstruction
Theekapun Charoenpong · Institutional Repositories DataBase (IRDB) · 2008
Due to a problem of current research occurring when recognizing facial expressions from a 2.5D partial face data set taken from any viewpoint ranging from -45 degrees to +45 degrees, we propose a new effective method for recognizing facial expressions based on 3D face reconstruction. This proposed method consists of two parts. In the first part, a 3D face is reconstructed by using an ellipse fitting technique. A face vector is used to detect a symmetry plane. By using the symmetry plane and real face data, the 3D face can be reconstructed. In the second part, facial expression is represented in terms of the change of crossing points on a face plane. Two crossing point analysis methods consisting of a crossing point distribution method and a displacement vector method are used to analyze distribution of crossing points. By using a support vector machines method, facial expressions can be recognized. Four facial expressions which were used in experiments were neutral, anger, surprise and smiling. The results show the feasibility of the proposed method. In the chapter 2, a 3D face is reconstructed. The data from each cross section of the head is fitted by an ellipse fitting technique. In each cross section of the head data, a vector pointing from a center of the ellipse to the facial surface is defined as a semi-major axis vector. Based on the experiments of all viewpoints, each cross section shows the different average ellipse fitting error. According to this property, the semi-major axis vector under the condition that the average ellipse fitting error of its cross section is less than 1 mm tend to point to the surface near the nose. By averaging the semi-major axis vectors obtained from one viewpoint, a face vector can be calculated. A nose ridge detection algorithm is used to detect the nose tip and the nose ridge. Vectors which are the projection of the face vector in the cross sections in the nose region are defined as projected face vector. Points on the facial surface in each cross section are projected onto the projected face vector. The projected points from the nose tip and nose ridge always show farthest distance from the center of the ellipse. To detect the nose tip and the nose ridge, the projected face vector is corrected to the point which shown the farthest distance. By using the nose tip and the nose ridge points, two vectors are obtained. One is a nose ridge vector. This vector is computed by a Principle Components Analysis method (PCA). Another one is a nose vector. The nose 2 vector points from the center of the head to the average of the nose ridge point. By using the two vectors, a symmetry plane is defined. This symmetry plane is corrected by using a symmetry plane correction algorithm. The 3D face is then reconstructed. In the chapter 3, a facial expression is represented in a face plane. From a reconstructed 3D face, facial surface is extracted by using a L*a*b* color space. A face plane algorithm is used to represent a facial expression in terms of crossing points. A face plane is a virtual plane across the head. Four neighboring points, which are points on the facial surface, are used to compute a normalized vector. The normalized vectors point to the center of the head. By using the normalized vectors, the face plane is computed. A point on the face plane which a normalized vector points through is defined as crossing point. In each facial expression, the character of the crossing point distribution is different. The crossing point distribution of an expression face is compared with a neutral face from the same person for recognition. Two methods are used to analyse the crossing point distribution. The first one is a crossing point distribution method. The crossing points on a face plane are divided into 36(6x6) sub-areas. The numbers of the crossing points in 36 sub-areas are used for the facial expression recognition. The second one is a displacement vector method. The size of an expression face is normalized to the size of a neutral face. Point on the facial surface with 3D-coordinate system is then converted to 2D-coordinate system. Two points on two faces which have the same 2D-coordinate are used to define a displacement vector. A displacement vector is a vector which points from a crossing point of a neutral face to a crossing point of an expression face. The two crossing points correspond to the pair point on the two faces. A surface on the expression face is divided to into 36(6x6) sub-areas based on the size of the neutral face. Displacement vectors in a sub-area are represented by an average of the displacement vectors. The 36 average displacement vectors are used for facial expression recognition. In the chapter 4, experiments were done. The head pose variation in yaw and two situations of the head ? with and without a cap were used. There were three experiments done. First, the nose ridge point was detected from a 2.5D partial face data set taken from any viewpoint ranging from -80 degrees to +80 degrees. The 2.5D partial face data were captured from 27 persons. All of the nose ridge from 223 samples with the cap can be 3 detected successfully. While ten samples fail in detection. Failure occurs because of the influence of messy hair. Second, the 3D face was reconstructed from a 2.5D partial face data set taken from any viewpoint ranging from -45 degrees to +45 degrees. The 2.5D partial face data were captured from 22 persons with and without a cap. A criterion defined by using a facial recognition was used to evaluate the reconstructed 3D face. By using the criterion, four samples of 187 samples were evaluated as an incorrect reconstruction. Third, the facial expressions were recognized by using a support vector machines technique and the radial basis function. 2.5D partial face data sets from 22 persons were taken from any viewpoint ranging from -45 degrees to +45 degrees. Four facial expressions which were smiling, surprise, anger and neutral were used. The experiments of the facial expression recognition were done based on the assumption that the person is known. The average recognition accuracy using the crossing point distribution scheme and the displacement vector scheme were 62.6% and 78.9% respectively. The advantages of our proposed method is that only one 2.5D partial face data set captured from any viewpoint of the face rotation between -45 degrees and + 45 degrees can be used. The results show the feasibility of the proposed method. In the chapter 5, the results of the proposed method were concluded.