ADQuaTe: An Automated Data Quality Test Approach for Constraint Discovery and Fault Detection
Hajar Homayouni, Sudipto Ghosh, Indrakshi Ray · 2019
Data quality tests validate the data stored in databases and data warehouses to detect violations of syntactic and semantic constraints. Domain experts grapple with the issues related to the capturing of all the important constraints and checking that they are satisfied. Domain experts often define the constraints in an ad hoc manner based on their knowledge of the application domain and needs of the stakeholders. We propose ADQuaTe, which is an automated data quality test approach that uses an unsupervised machine learning technique to discover constraints that may have been missed by experts. ADQuaTe marks records that violate the constraints as suspicious and explains the violations. We evaluate ADQuaTe on real-world applications using a health data warehouse and a plant diagnosis database to demonstrate that the approach can uncover previously detected as well as new faults in the data.