An Evaluation of Large Language Models for Geological Named Entity Recognition
Rafael Oleques Nunes, André Suslik Spritzer, Dennis Giovani Balreira, Carla Maria Dal Sasso Freitas, Joel Luís Carbonera · 2024
Recent advancements in Natural Language Processing (NLP) have highlighted the success of transformer-based models in various tasks, including Named Entity Recognition (NER). This study evaluates and compares these models using GeoCorpus-3, Portuguese's most extensive annotated geological corpus. We investigate the performance of BERTimbau and XLM-RoBERTa models across 30 entity classes, incorporating both linear and CRF layers, and assess their effectiveness using micro, macro, and weighted F1-scores. Our results reveal that transformer models, particularly XLM-RoBERTa, achieve superior performance compared to previous methods. Statistical analysis indicates significant performance differences, with BERTimbau-Linear showing lower performance than the other models. This study provides a comprehensive evaluation of transformer models on a geological corpus and makes the updated corpus and models publicly available to advance research and development in this domain.