Polish Coreference Corpus as an LLM Testbed: Evaluating Coreference Resolution within Instruction-Following Language Models by Instruction–Answer Alignment
Karol Saputa, Angelika Peljak-Łapińska, Maciej Ogrodniczuk · 2024
In this article, we analyse coreference resolution in encoder-and decoder-based approaches in the Polish language.We convert the Polish Coreference Corpus into the instructions suitable for training language models and create supplementary data based on examples that are difficult for encoder-based models, analyse them and create additional questions for more precise mention boundary detection and other ambiguities found.We propose an evaluation framework for our instructions.The best closed model, Claude 3 Sonnet, achieves 44.52 CoNLL F 1 in instruction following, zero-shot setting, which is surpassed by the fine-tuned Llama 3.1 8B model, which achieves 46.54 F 1 .