Dynamic Knowledge Graph Evaluation
Roos M. Bakker, Maaike de Boer · 2024
In a world where information is exchanged at an increasing pace, knowledge becomes quickly outdated. Formal constructs that capture human knowledge, such as knowledge graphs and ontologies, need to be updated and evaluated to stay relevant and functioning. However, evaluating knowledge models is labour-intensive and prone to errors. This study addresses the challenge of automatically evaluating changes in existing knowledge graphs. We introduce syntactic and semantic metrics tailored for change evaluation. The metrics are implemented and validated through experiments on knowledge graphs across various domains. In these experiments, real-world changes are simulated by removing concepts and introducing faulty ones before measuring the quality with the syntactic and semantic metrics. The hypothesis is that such changes decrease the quality of the knowledge graph: removing concepts influences syntactic qualities such as the structure of the model, while adding faulty concepts affects semantic qualities like model consistency. The validation results support this hypothesis, demonstrating that the introduced metrics effectively reflect the changes made to the graph. Additionally, the experiments show that the size and domain specialisation of a knowledge graph influence how well the metrics detect changes. Overall, this study proposes a novel set of evaluation metrics and provides evidence of their effectiveness for assessing modifications to knowledge graphs across different domains. These metrics can help developers detect errors, highlight unintended side effects, and flag other quality changes that might otherwise go unnoticed.