Direct Density-Ratio Estimation with Dimensionality Reduction
Masashi Sugiyama, Taiji Suzuki, Takafumi Kanamori · Cambridge University Press eBooks · 2012
The approaches of direct density-ratio estimation explained in the previous chapters were shown to be promising in experiments with naive kernel density estimation in experiments. However, these methods still perform rather poorly when the dimensionality of the data domain is high. The purpose of this chapter is to introduce ideas for mitigating this weakness, following Sugiyama et al. (2010a, 2011b). A basic assumption behind the approaches explained here is that the difference between the two distributions in the density ratio (i.e., the distributions corresponding to the numerator and denominator of the density ratio) does not spread over the entire data domain, but is confined in a low-dimensional subspace – which we refer to as the heterodistributional subspace . Once the heterodistributional subspace can be identified, the density ratio is estimated only within this subspace. This will lead to more stable and reliable estimations of density ratios. Such an approach is called direct density-ratio estimation with dimensionality reduction (D 3 ; pronounced “D-cube”). In this chapter, two approaches to D 3 are described. In Section 8.1, a heuristic method based on discriminant analysis is explained. This method is shown to be computationally very efficient, and thus is very practical. On the other hand, in Section 8.2, a more theory-oriented approach based on divergence maximization is introduced. This method is justifiable under general settings, and thus it has a wider applicability. Numerical examples are shown in Section 8.3, and the chapter is concluded in Section 8.4.