CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification
Matúš Falis, Hang Dong, Alexandra Birch, Beatrice Alex · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021
Large-Scale Multi-Label Text Classification (LMTC) includes tasks with hierarchical label spaces, such as automatic assignment of ICD-9 codes to discharge summaries.Performance of models in prior art is evaluated with standard precision, recall, and F 1 measures without regard for the rich hierarchical structure.In this work we argue for hierarchical evaluation of the predictions of neural LMTC models.With the example of the ICD-9 ontology we describe a structural issue in the representation of the structured label space in prior art, and propose an alternative representation based on the depth of the ontology.We propose a set of metrics for hierarchical evaluation using the depthbased representation.We compare the evaluation scores from the proposed metrics with previously used metrics on prior art LMTC models for ICD-9 coding in MIMIC-III.We also propose further avenues of research involving the proposed ontological representation.