Understanding Language Model from Questions in Social Studies for Students
Kaito Kawashima, Saneyasu Yamaguchi · 2021 IEEE International Conference on Big Data (Big Data) · 2021
Artificial intelligence, especially artificial intelligence based on deep neural networks, has improved significantly. In particular, great progress has been made in natural language processing and image recognition. In the natural language processing field, many techniques and methods, such as transformer, attention, and self-attention, have been proposed and have improved this field. Recently, BERT and RoBERTa are expected to be some of the most promising natural processing technologies. Through deep learning has achieved high accuracy in various fields such as natural language processing, it has been pointed out that it lacks interpretability and explainability for decision. For addressing this issue, providing interpretability and explainable AI have been studied. For BERT and RoBERTa, discussions on understanding what language models know about language have been studied. In this paper, we discuss what the pre-trained Japanese BERT and RoBERTa language models know. We solved masked questions for Japanese students using the deep learning artificial intelligence using these language models and investigated their knowledge and the dependence of their accuracies on the domain.