Sharing is CAIRing: Characterizing principles and assessing properties of universal privacy evaluation for synthetic tabular data
Tobias Hyrup, Anton D. Lautrup, Arthur Zimek, Peter Schneider–Kamp · Machine Learning with Applications · 2024
Data sharing is a necessity for innovative progress in many domains, especially in healthcare. However, the ability to share data is hindered by regulations protecting the privacy of natural persons. Synthetic tabular data provide a promising solution to address data sharing difficulties but does not inherently guarantee privacy. Still, there is a lack of agreement on appropriate methods for assessing the privacy-preserving capabilities of synthetic data, making it difficult to compare results across studies. To the best of our knowledge, this is the first work to identify properties that constitute good universal privacy evaluation metrics for synthetic tabular data. The goal of universally applicable metrics is to enable comparability across studies and to allow non-technical stakeholders to understand how privacy is protected. We identify four principles for the assessment of metrics: Comparability, Applicability, Interpretability, and Representativeness (CAIR). To quantify and rank the degree to which evaluation metrics conform to the CAIR principles, we design a rubric using a scale of 1–4. Each of the four properties is scored on four parameters, yielding 16 total dimensions. We study the applicability and usefulness of the CAIR principles and rubric by assessing a selection of metrics popular in other studies. The results provide granular insights into the strengths and weaknesses of existing metrics that not only rank the metrics but highlight areas of potential improvements. We expect that the CAIR principles will foster agreement among researchers and organizations on which universal privacy evaluation metrics are appropriate for synthetic tabular data. • Characterizing four principles for good universal privacy metrics for synthetic data. • Comparability, Applicability, Interpretability, and Representativeness (CAIR). • We assess privacy metrics using a rubric-based method with at least one evaluator. • Universal privacy metrics allow stakeholders to agree on common evaluation methods. • CAIR assessments provide granular insights into strengths and weaknesses of metrics.