ARIES: A General Benchmark for Argument Relation Identification
Debela Tesfaye Gemechu, Ramon Ruiz-Dolz, Chris Reed · 2024
Measuring advances in argument mining is one of the main challenges in the area.Different theories of argument, heterogeneous annotations, and a varied set of argumentation domains make it difficult to contextualise and understand the results reported in different work from a general perspective.In this paper, we present ARIES, a general benchmark for Argument Relation Identification aimed at providing a standard evaluation for argument mining research.We evaluated three different architectures for Argument Relation Identification on ARIES: sequence and token modelling, and sequence-to-sequence alignment, together with the three main Transformer-based model architectures: encoder-only, decoder-only, and encoder-decoder.Furthermore, the benchmark consists of eight different argument mining datasets, covering the most common argumentation domains, and standardised with the same annotation structures.This paper provides a first comprehensive and comparative set of results in argument mining across a broad range of configurations to compare with, both advancing the state-of-the-art, and establishing a standard way to measure future advances in the area.Across varied task setups and architectures, our experiments reveal consistent challenges in cross-dataset evaluation, with notably poor results.Given the models' struggle to acquire transferable skills, the task remains challenging, opening avenues for future research.