QCET: An Interactive Taxonomy of Quality Criteria for Comparable and Repeatable Evaluation of NLP Systems
Anja Belz, Simon Mille, Craig Thomson, Rudali Huidrom · 2024
Four years on from two papers (Belz et al., 2020;Howcroft et al., 2020) that first called out lack of standardisation and comparability in quality criteria assessed in NLP system evaluations, researchers still use widely differing quality criteria names and definitions, meaning that it continues to be unclear when the same aspect of quality is being assessed in two evaluations.While normalised quality criteria were proposed at the time, the list was unwieldy and using it came with a steep learning curve.In this demo paper, our aim is to address these issues with an interactive taxonomy tool that enables quick perusal and selection of the standardised quality criteria, and provides decision support and examples of use at each node.