INFORMATION THEORETIC APPROACHES TO RINCIPAL AND INDEPENDENT COMPONENT ANALYSIS: A SIMPLIFIED VIEW
Dinesh Kumar · 2008
Principal Component Analysis (PCA) and Independent Component Analysis (ICA) are the techniques that deal with extracting the independent components from linear mixtures of Gaussian and non-Gaussian data at the input respectively. PCA is a classical method that deals with the second order statistics of data. It is also known as Karhunen-Loeve Transform or the Hotelling Transform in some application areas. ICA is a generalization of PCA that takes into account the higher order statistics also. This paper presents a simplified view of information theoretic approaches to the problem of Principal Component and Independent Component Analysis. With the help of these techniques we find a linear representation of multivariate data so that the components are as statistically independent as possible. Such representations capture essential features of the data in many applications. This paper summarizes major approaches that are based on information theoretic concepts and includes the applications of ICA.