Fairness Testing of Machine Learning Models using Combinatorial Testing in Latent Space
Arjun Dahal, Sunny Shree, Yu Lei, Raghu N. Kacker, D. Richard Kuhn · 2025
Decision-making by Machine Learning (ML) models can exhibit biased behavior, resulting in unfair outcomes. Testing ML models for such biases is essential to ensure unbiased decision-making. In this paper, we propose a combinatorial testing-based approach in the latent space of a generative model to generate instances that assess the fairness of black-box ML models. Our approach involves a two-step process: generating t-way test cases in the latent space of a Variational AutoEncoder and performing fairness testing using the instances reconstructed from these test cases. We experimentally evaluated our approach against an approach that generates t-way instances in the input space for fairness testing. The results indicate that the latent-space approach produces more natural test cases while detecting the first fairness violation faster and achieving a higher ratio of discriminatory instances to the total number of generated instances.