Feature Extraction in Kernel Space Using Bhattacharyya Distance as Criterion Function
Jian Xia · Chinese Journal of Computers · 2004
Authors propose a novel approach to feature extraction for classification in kernel space using Bhattacharyya distance, determining the upper-bond of Bayes error, as criterion function, which is called BKFE. The key idea of BKFE is that the data are nonlinearly mapped into high dimensional kernel space at first. Then we can find a set of discriminantly informative features in kernel space to linearly map the data into low dimensional feature space, where the Bhattacharyya distances between classes are maximized. First authors draw on kernel theory and nonlinear optimization technique to develop BKFE for binary classification problem, and solves the feature extraction problem by quadratic programming method, which endows BKFE with fast and global convergence. Then authors extend BKFE to multi-class classification problem. After defining the kernel space feature extraction matrix (KFEM), they discuss theoretically in some depth the relationship exiting between KFEM and intrinsic discriminant dimension of a classification problem, gives Theorem 1 for this relationship. According to theorem 1, the rank of KFEM is equal to the intrinsic discriminant dimension, and its eigenvectors with non-zero eigenvalue are discriminantly informative features. Because of high dimension, the rank and eigenvectors of KFEM are difficult to calculate directly. In order to overcome the difficulty, authors construct a new low dimension kernel matrix, which has the same rank as KFEM, and extract discriminantly informative features by calculating the eigenvectors of the new kernel matrix. In particular, the upper-bond of intrinsic discriminant dimension for a classification problem is M(M-1)/2 (M is class number). It is important because it indicates how many features should be extracted for a given classification problem. Compared with KPCA (Kernel Principal Components Analysis), KFD (Kernel Fisher Discriminate), and FD (Fisher Discriminate), BKFE has two desirable advantages: (1) features extracted by BKFE are more effective for classification; (2) it predicts the upper-bound of the number of necessary features to achieve the same classification accuracy as in the original space for a given pattern recognition problem. Experimental results show that BKFE can provide more informative features for pattern classification than others.