SKH-NLP at LLMs4OL 2024 Task B: Taxonomy Discovery in Ontologies Using BERT and LLaMA 3
Seyed Mohammad Hossein Hashemi, Mostafa Karimi Manesh, Mehrnoush Shamsfard · Open Conference Proceedings · 2024
Taxonomy discovery in ontologies refers to extracting the parent class from the child class. By modeling this task as a classification problem, we addressed it using two different approaches. The first approach involved fine-tuning the “BERT-Large” model with various prompts and using it in a classification system. In the second approach, we utilized the “LLaMA 3 70B” model, experimenting with different prompts and modifying them to achieve the best results. Additionally, we evaluated the correctness of the answers using substring and Levenshtein distance functions. The results indicate that, with appropriate fine-tuning, the BERT model can achieve performance levels comparable to those of more recent and significantly larger language models, such as LLaMA 3 70B. However, with appropriate prompts, LLaMA 3 70B performs slightly better than BERT, highlighting the importance of prompt quality. Ultimately, further experiments on different settings for fine-tuning BERT, few-shot learning, and using knowledge graphs for validating the model's answers for LLaMA are recommended to improve the results. Additionally, testing other models and examining the results of various encoder-based and decoder-based models can be employed.